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"We  are  deceiving  ourselves  if  we  naively  believe  that  we  are  not  as  human  as  the  people 
around  us  and  that  we  do  not  tend  to  aim  opportunistically  for  conclusions  that  fit 
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valuations  and  by  assigning  prime  importance  to  observed  facts,  we  only  partly  purge 
these  biases  from  our  mind.  ” 

(G.  Murdal,  Objectivity  in  Social  Research,  1969,  p.  43) 


This  research  was  funded  by  the  Defense  Systems  Management  College,  Ft.  Belvoir, 
VA. 


Decision  Analysis  In  Source  Selection 


ABSTRACT 

This  research  investigates  the  efficacy  of  applying  formal 
decision  analysis  methods  to  the  selection  of  the  '>best”  contractor 
for  defense  systems  acquisition.  Specifically,  Multiattribute 
Utility  Theory  (MAUT)  and  the  Analytic  Hierarchy  Process  (AHP)  are 
applied  to  source  selection  within  the  United  States  Air  Force. 
The  source  selection  process  is  described.  MAUT  and  AHP  are 
applied  to  an  actual  Air  Force  source  selection  case  study.  We 
conclude  the  Air  Force  source  selection  process  can  benefit  from 
the  application  of  decision  analysis. 


1.  INTRODUCTION 

The  defense  budget  increased  significantly  during  the  Reagan 
Administration,  resulting  in  enormous  expenditures  for  weapon 
systems.  Along  with  these  increased  defense  expenditures  came  many 
reports  of  fraud,  waste,  abuse,  and  inefficiency.  Reports  of 
$600.00  toilet  seats  and  $120.00  hammers  outraged  the  American 
public  and  prompted  close  scrutiny  of  the  acquisition  process.  In 
July,  1985  President  Reagan  commissioned  a  Blue  Ribbon  Panel  to 
investigate,  improve  and  restore  public  confidence  in  the  weapon 
systems  acquisition  process.  The  Department  of  Defense  is  now 
implementing  most  of  the  recommendations  of  the  "Defense  Management 
Review"  performed  by  this  Panel,  formally  called  the  Packard 
Commission. 

The  Packard  Commission  studied  and  recommended  improvements  in 
acquisition  organization  and  procedures,  building  prototypes  and 
testing,  cost  and  performance  measures,  program  stability, 
competition,  management  of  acquisition  personnel,  and  government- 
industry  accountability  (Kovacic  1987) .  There  is  however,  yet 
another  aspect  of  the  systems  acquisition  process  which  should  be 
studied  and  improved:  decision  making  by  acquisition  personnel. 
This  paper  investigates  how  decision  making  might  be  improved  by 
the  use  of  formal  decision  analysis  methodologies.  In  particular, 
we  explore  how  Multi-Attribute  Utility  Theory  (MAUT)  and  the 
Analytic  Hierarchy  Process  (AHP)  might  be  employed  to  improve  the 
source  selection  phase  of  the  weapon  systems  acquisition  process. 
Source  selection  is  that  part  of  the  acquisition  process  in  which 
the  contractor  or  supplier  of  the  weapon  system  is  chosen. 

1.1  NEED  FOR  THIS  RESEARCH 

The  source  selection  procedures  which  must  be  followed  are 
specified  in  Department  of  Defense  Directives,  Federal  Acquisition 
Regulations,  and  regulations  and  directives  imposed  by  the 
individual  services  and  major  commands,  (c.f.  Air  Force  Regulation 
70-15)  This  study  is  concerned  only  with  the  source  selection 
process  employed  within  Systems  Command  of  the  United  States  Air 
Force. 
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There  are  several  aspects  of  the  formal  source  selectipn 
process  which  are  congruent  with  the  established  decision  analysis 
methodologies  mentioned  above.  In  .particular,  the  igurrent  source 
■selection  process  requires  that: 

1.  the  criteria  for  the  source  iselection  decision  be 
clearly  specified, 

2.  the  relative  ipriority  of  the  criteria  be  established, 

-3.  the  attributes  which  measure  how  well  cpntractor 
proposals  meet  specified  requirements  be  defined, 

4,  the  proposals  be  evaluated  on  the  basis  of  these 
attributes  and  criteria,  and 

5.  the  "best”  proposal  be  selected  based  upon  the 
analysis  conducted  in  steps  1  through  4,  above. 

It  is  in  steps  4  and  5  where  differences  emerge  between 
current  source  selection  practices  and  established  decision 
analysis  methods.  Whereas  MAUT  would  recommend  selection  of  the 
contractor  whose  proposal  would  "maximize  expected  utility"  and  AHP 
would  recommend  selection  of  the  contractor  whose  proposal  has  the 
highest  "overall  composite  priority," 

the  current  sohrce  selection  process  provides  no  guidance 
as  to  how  judgements  and  evaluations  performed  in  steps  1 
through  4  should  be  synthesized  to  yield  the  preferred 
alternative. 

This  issue  is  the  central  focus  of  this  research:  Can  decision 
analysis  improve  source  selection  decisions,  especially  in  the 
synthesis  of  judgements  and  evaluations  to  yield  a  ranking  of 
contractors? 

The  need  for  a  thorough,  consistent  and  rational  analysis  is 
clearly  recognized  in  the  directives  and  regulations  which  guide 
the  source  selection  process.  As  stated  in  Air  Force  Regulation 
70-15/AFFARS  Appendix  AA: 

The  principal  objective  of  the  major  source  selection 
process  is  to  select  the  source  whose  proposal  has  the 
highest  degree  of  credibility  and  whose  performance  can 
be  expected  to  best  meet  the  government's  requirements  at 
an  affordable  cost.  The  process  must  provide  an 
impartial,  equitable,  and  comprehensive  evaluation  of  the 
competitors'  proposals  and  related  capabilities.  The 
process  should  be  accomplished  with  minimum  complexity 
and  maximum  efficiency  and  effectiveness.  It  should  be 
structured  to  properly  balance  technical,  financial,  and 
economic  or  business  considerations  consistent  with  the 
phase  of  the  'acquisition,  program  requirements,  and 
business  and  legal  constraints.  It  must  be  sufficiently 
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flexible  to  accommodate  the  objectives  of  th-a  acquisition 
and  a  decision  must  'be  compatible  vdth  progrtim 
requirements,  risks,  and  conditions,  (p.  ?,-) 

The  Federal  Acquisition  Regulation  regarding  scarce  selection 
(FAR  SUBPART  15.6)  specifies  similar  objectives: 

Source  selection  procedures  are  designed  to: 

(a)  Maximize  competition; 

(b)  Minimize  the  complexity  of  the  solicitation, 
evaluation,  and  the  selection  decision; 

(c)  Ensure  impartial  and  comprehensive  evaluation  of 
contractors'  proposals;  and 

(d)  Ensure  selection  of  the  source  whose  proposal  has  the 
highest  degree  of  realism  and  whose  performance  is 
expected  to  best  meet  stated  Government  requirements. 

(pp.  1-2.) 

Thus,  it  seems  the  source  selection  process  could  benefit 
substantially  from  decision  analysis. 

1.2  PREVIOUS  RESEARCH 

This  research  is  not  the  first  to  investigate  this  issue. 
Goren  (1981)  identified  the  following  "major  problems, 
deficiencies,  and  difficulties"  with  the  source  selection  process: 

1.  Difficulties  in  circumscribing  the  decision  factors 
and  consolidating  them  all  for  a  final  decision.  This 
stems  from  the  complex,  multi-objective  nature  of  the 
acquisition  decisions. 

2.  Lack  of  balance  of  political  impact  of  individuals, 
groups  and  organizations,  imposed  upon  the  decision¬ 
making  process.  Disorder  in  the  representation  of  those 
who  have  stakes  in  the  decision. 

3.  Existence  of  phenomena  such  as  biased  decision 
presentation,  which  is  amplified  by  the  inevitable 
reliance  of  top-level  decision  makers  on  their 
subordinates'  evaluations. 

4 .  Aversion  to  the  usage  of  quantitative  techniques  as 
top-level  decision  aids  which  stems  from  the  lack  of 
confidence  in  the  capability  of  these  techniques  to 
incorporate  intuitive,  jud^ental  issues,  (p.  130) 

These  deficiencies  were  ascertained,  in  large  part,  through  an 
opinion  survey  and  interviews  of  those  working  in  the  systems 
acquisition  field. 
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Goren  concludes  that: 

1.  The  MAUT  procedure  should  be,  used  as  the  maqor  tool 
for  presenting  and  evaluating  alternatiyes  for 
acquisition  decisions  at  the  top  level. 

2.  For  each  major  system,  MAUT  should  serve  as  a 
comprehensive,  ongoing,  updated,  evaluation  framework 
through  the  whole  acquisitibn  life  cycle,  (pp.  132-133) 

He  reached  these  conclusions  despite  the  fact  that  he  did  not 
attempt  to  actually  apply  MAUT  to  source  selection,  nor  did  he 
construct  a  realistic  MAUT  model. 

Donnell  and  Ulvila  (1980)  conducted  a  decision  analysis  of  the 
Advanced  Scout  Helicopter  (ASH)  candidates  for  the  U.S.  Army.  They 
assisted  the  ASH  Special  Study  Group  which  was  tasked  with 
selecting  an  effective  and  affordable  ASH  and  demonstrating  that 
the  selected  ASH  was  the  most  cost  effective  of  the  candidates. 

They  accomplished  this  by  constructing  an  evaluation  model 
designed  to  capture  objective  as  well  as  subjective  factors 
relevant  to  the  decision  at  hand.  There  were  thirteen  candidates 
and  over  seventy  relevant  attributes  to  be  considered.  The  output 
of  the  resultant  MAUT  model  was  a  "numerical  representation  of  the 
worth  of  each  ASH  candidate."  (p.  1)  They  provided  a  detailed 
discussion  of  the  model  and  its  development. 

Donnell  and  Ulvila  did  not,  recommend  a  final  ASH  because  their 
model  was  only  an  initial  attempt  to  structure  the  problem.  They 
felt  their  model  required  refinement  in  order  for  them  to  reach  a 
final  recommendation.  They  noted,  however,  that  the  modeling 
process  allowed  the  Special  Study  Group  "to  more  fully  understand 
tradeoffs  of  the  competing  decision-related  variables."  (p.  89) 
They  also  pointed  out  the  benefits  of  modeling  this  problem  using 
MAUT: 


"...all  numerical  assessments  (were)  supported  through 
written  rationale,  and  it  (was)  possible  to  vary  any 
score  or  weight  that  may  be  in  question  and  to  determine 
the  impact  of  such  variations  on  the  result. 

Moore  and  Neve  (1987)  employed  the  AHP  to  determine  the 
relative  importance  of  criteria  in  the  selection  of  contractors  for 
the  construction  of  DoD  facilities.  They  listed  relevant  criteria 
which,  they  contend,  are  generally  agreed  upon  by  the  "construction 
community . "  They  then 

...queried  DoD  facility  construction,  procur ''ment , 
construction  (sic) ,  and  engineering  experts  from  tne  Air 
Force,  Army,  and  Navy  to  determine  which  factors  they 
felt  to  be  important  on  the  basis  of  their  judgement  and 
experience. . .The  intent  of  the  ranking  was  to  establish  a 
sample  weighting  that  could  be  used  as  a  starting  point 
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for  developing  evaluation  and  selection  plans... The 
results  of  the  rank  ordering  displayed  some  variance,  but 
were  statistically  significant  at  the  95%  level.  With 
few  notable  exceptions,  engineering,  procurement,  and 
construction  experts  agreed  on  the  importance  of  the 
variables,  (pp.  B-1  -  B--2) 

Thus,  the  AHP  was  used  only  to  ascertain  the  relative 
importance  of  the  generally  agreed  upon  criteria.  No  discussion  is 
provided  about  building  a  complete  hierarchical  decision  model  and 
conducting  a  complete  AHP  analysis  to  recommend  a  preferred 
contractor. 

Cook  (1986,  1987,  1989)  has  done  considerable  work  in  applying 
AHP  to  source  selection.  He  (1986)  used  AHP  to 

...help  contracting  and  program  managers  identify  and 
evaluate  the  relevant  criteria,  subcriteria,  and 
alternatives  to  be  considered  in  formulating  a 
comprehensive  strategy,  and  to  enable  these  professionals 
to  integrate  their  experience,  judgement,  and  the  facts 
that  are  available  into  a  coherent,  comprehensive 
competition  strategy,  (p.  273) 

Cook  (1986)  applied  the  AHP,  as  implemented  in  the  software 
package  Expert  Choice  (1983)  to  select  a  contractor  foj.  an  Air 
Force  fighter  aircraft  engine.  He  proved  the  feasibility  of 
applying  this  decision  analysis  methodology  to  such  problems.  He 
concludes: 


Is  there  a  need  for  Expert  Support  Systems  in  the 
government  procurement  arena?  The  answer  seems  to  be 
yes,  at  least  in  the  complex  competition  strategy 
formulation  process.  The  decision  is  based  on  a  number 
of  criteria  and  subcriteria,  and  offers  several 
alternative  techniques  for  introducing  competition  into 
government  procurement . 

This  paper  has  presented  an  ESS  model,  based  on  the 
Analytic  Hierarchy  Process  and  using  Expert  Choice,  that 
enables  the  decision  makers  to  organize  and  structure 
their  judgements.  It  facilitates  the  innate  decision 
process  we  use  in  simple  decisions  and  allows  that  logic 
to  be  applied  to  complex  decisions.  It  enables  the 
decision  maker  to  perform  "what  if"  exercises  to  quickly 
trace  the  effect  of  changes  in  judgement  or  in  conditions 
outside  his  control.  Finally,  Expert  Choice  and  the 
model  are  easy  to  use,  require  little  training  time  for 
initial  use,  and  run  on  the  most  popular  microcomputers. 

(pp.  286-287) 

Cook  (1989)  also  used  the  AHP  to  assist  in  a  decision 
regarding  "component  breakout"  to  encourage  competition  in  the 
procurement  of  major  weapon  systems.  "Component  breakout"  is 
separating  major  components  from  the  prime  contract  and  obtaining 
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those  in  separate  procurement  actions.  Thus,  each,  component,  is; 
subjected  to  the>  acquisition  process  separately  thereby  encouraging 
competition  and  "multiple  sources."  His  paper  chronicles  an  actual 
case  study  for  the^  procurement  of  the'  F-is  fighter  -aircraft  radar 
system i  He  concludes: 

1.  The  decision  itself  was  transformed  from,  unstructured 
to  structured. 

2,.  The  DSS  increased  group  and  management  confidence  in 
the  decision  process  and  in  the  group  recommendation. 

3.  The  decision  took  less  time,  in  the  opinion  of  the 
group  members,  than  would  have  been  required  using  an 
alternative  methodology. 

4.  As  a  result  of  the  imbedded  documentation  and  audit 
crail,  the  team  was  able  to  present  its  findings  and 
recommendations  in  a  rational  manner  when  briefing  the 
program'  director,  thus  winning  the  director's  approval. 

(pp.  11-13) 

Vickery  (1989)  developed  a  fictitious  source  selection  case 
and  asked  a  number  of  individuals  to  analyze  the  problem  and  decide 
which  contractor  should  be  awarded  the  contract.  One  group  was 
introduced  to  AHP  and  the  other  was  not.  The  quality  of  their 
decision  was  judged  according  to  six  criteria:  effectiveness, 
consistency,  speed,  difficulty,  understanding,  and  confidence. 
While  the  AHP  group  reported  significantly  higher  "speed"  and 
"understanding,"  and  significantly  lower  "difficulty,"  there  was  no 
significant  difference  in  decision  "effectiveness"  or 
"consistence."  Vickery  offers  several  plausible  explanations  of 
why  this  could  be  the  case,  including'  the  possibility  the  case  was 
too  simple.  One  interesting  result  was  every  person  in  the  control 
group  used  an  ad  hoc  method  which  was  similar  to  the  AHP,  lending 
credence  to  the  theory  that  the  AHP  "facilitates  the  innate 
decision  making  process"  as  suggested  by  Cook  (1986) . 


1.3  Conclusions  and  Recommendations  of  This  Research 

It  is  the  conclusion  of  this  research  that  formal  decision 
analysis  should  be  used  to  improve  the  source  selection  process. 
In  particular,  decision  analysis  would  assist  evaluators  in 
structuring  this  important  problem,  ascertaining  the  criteria  upon 
which  the  decision  should  be  based,  deriving  weights  for  these 
criteria,  and  evaluating  contractor  proposals  against  these 
criteria.  Moreover,  decision  analysis  would  facilitate  a  more 
logical  and  consistent  synthesis  of  judgements  into  a  final  ranking 
of  contractor  proposals. 
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1.4  organization  of  the  Paper 

In  Section  2,  we  describe  the  current  U.S.  Air  Force  source 
selection  process  to  include  organizational  structure,  individuals 
involved  and  their  responsibilities,  stages  of  the  source  selection 
process  with  emphasis  on  the  evaluation  and  decision  methods. 
Advantages  and  disadvantages  of  the  current  process  are  discussed, 
and  areas  for  improvement  are  identified.  An  actual  case  study  is 
introduced  which  provides  the  basis  for  the  MAUT  and  AHP  models 
developed  in  Sections  4  and  5.  Section  3  discusses  decision 
analysis  with  emphasis  on  the  fundamental  process,  advantages  and 
disadvantages.  In  Section  4,  MAUT  is  described,  applied  to  the 
actual  case,  and  its  deficiencies  and  advantages  noted.  Section  5 
provides  an  analogous  discussion  and  application  of  the  AHP. 
Finally,  conclusions  and  recommendations  for  further  research  are 
discussed  in  Section  6. 
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2.  CURRENT  SOURCE  SELECTION  PROCESS 
2.1  The.  Magnitude  and  Importance  of.  the  Source.  Selection  Problem 

Source  selection  involves  billions  of  dollars  each  year  and 
employs  thousands  of  people  within  the  Department  of  Defense.  In 
fiscal  year  1988,  within  Air  Force  Systems  Command  alone,  there 
were,  approximately  15,000  contracting  actions  worth  about  $23 
billion.  These  actions  were,  divided  among  the  product;  divisions  as 
illustrated  in  Figures  1  and  2. 


ASD  7455 


TOTAL:  14,883 


ASD:  Aeronautical  Systefns  Division 
AFFTC:  Air  Force  Flight  Test  Center 
BSD:  Ballistic  Systerns  Division 
AFSTC;  Air  Force  Space  Test  Center 
HSD:  Munitions  Systems  Division 
BSD:  Electronic  Systems  Division 
ESHC:  Eastern  Space  and  Missile  Center 
SSU:  Space  Systems  Division 
USMC:  Western  Space  and  Missile  Center 

FIGURE  1:  Number  of  Contracting  Actions, 
U.S,  Air  Force  Systems  Command,  FY  1988 
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V 


ASD  12559.3 


TOTAL:  22922.3 


FIGURE  2;  Value  of  Contracting  Actions, 
u.s.  Air  Force  Systems  Command,  FY  1988 


2.2  Organization  of  Source  Selection 
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".The  -Air  Eorce  has  .  outlined  the  :systenis  , acquisition  ^.process  and 
■procedures  .in  .regulations  ..and  -directives.  .Of  ^particular  ..interest 
to  this  x-esearch  .is  Air  .Force  .Regulation  "70t-i-5  .  A[ER  »736-.l'5/'AFFARS, 
Appendix  .AA,  rdescribes  a  “•ty.pical  =  source  ..selection  organizational 
.structure .  .See  .Figure  3 . 


ADVISORS 


FIGURE  3 :  Source  Selection  .Organization 


Source  selection  activity  is  -structured  hierarchically  which 
provides  the  depth  and  the  breadth  needed  'to  .evaluate  the  merits  of 
the  contract  proposals.  The  ..Source  .Selection  Authority  (SSA)  is 
the  individual  who  will  .decide  the  contract  ^winner.  He/she  has 
available  the  guidance,  advice  .and  expertise  of  the  Source 
Selection  Advisory  Council  (:SSAC) .  .The  SSAC  is  composed  of  senior 
government  personnel  with  extensive  source  .selection  experience. 

The  Source  Selection  Evaluation  .Board  (SSEB)  however,  conducts 
the  detailed  evaluations.  The  SSEB  is  divided  into  "Item  Teams," 
which-  comprise  the  SSEB.  These  are  .charged  with  evaluation  of  a 
specific  area  of  the  contract  proposals.  Each  Item  Team  contains 
several  members,  each  with  expertise  and  .experience  relevant  to  the 
team's  assigned  area  of  responsibility. 
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The  System  Program  Office  (SPO)  manages  the  entire  acquisition 
process  and  oversees  contractor  performance.  The  SPO. of ten  employs 
a  ’'matrix"  organizational  structure  to  obtain  the  technical 
services  and  expertise  of  others  not  formally  assigned  to  the  SPO. 


2.3  A  Brief  Overview  of  Source  Selection 

Once  the  Source  Selection  Plan  (SSP)  is  developed  by  the  SPO, 
with  the  advisement  of  the  Business  Strategy  Panel,  and  approved  by 
the  Source  Selection  Authority,  the  SPO  prepares  a  final 
solicitation  for  potential  contractors  who  intend  to  bid  .for  the 
contract.  Prospective  contractors  respond  to  the  government's 
solicitation  within  a  specified  time  frame  with  proposals  clearly 
describing  how  they  intend  to  meet  the  government's  expectations. 
When  the  government  receives  the  proposals,  the  source  selection 
has  officially  begun.  Figure  4  shows  a  time  line  for  a  typical 
source  selection. 

The  evaluation  of  the  proposals  is  perhaps  the  most  difficult 
aspect  of  source  selection,  and  is  certainly  the  most  subjective. 
A  myriad  of  technical,  cost  and  managerial  factors  are  considered 
in  the  source  selection  decision.  Technical,  cost  and  management 
experts  are  organized  into  "Item  Teams,"  each  tasked  with 
investigating  and  evaluating  a  specific  area  of  the  proposals. 
These  teams  evaluate  the  proposals  against  the  criteria  specified 
in  Source  Selection  Plan. 

The  evaluation  criteria  are  defined  as  precisely  and 
unambiguously  as  possible,  but  are  often  subject  to  contractor 
interpretation.  This  is  because  of  the  uncertainties  involved  with 
researching  and  developing  new  technologies.  As  a  result,  the  Item 
Teams  must  also  assess  the  risk  inherent  in  the  factors  they  are 
tasked  to  evaluate.  The  lack  of  operational  test  data  often 
necessitates  subjective  assessments  of  risk. 
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SOURCE  SELECTION  EVENTS' 


MONTHS; 


'iSi^  rtM 

$$A<  S*wft«$«(KV^  AwilMvitr 

fCOi  PrMvri»«l!4>«ir««UN«OnWr 
lAPUt  PifxuOffw 

SSAC(  5*Mfrf  A4»ifr7 

$SAi  SMfM  SvWftiM  AwtWtty 


FIGURE  Time  Line  for  Source  selection 


In  an  attempt  to  gauge  contractor  performance.,  each  Item  Team 
must  grade  the  contractor  against  the  standards  found  in  the 
original  solicitation »  As  the  example  from  AFR  70-15/AFFARS 
Appendix  AA  (Figure  5  below)  shows,  the  government  would  evaluate 
the  contractors'  proposals  against  the  following  "evaluation  (or 
assessment)  criteria:" 

1)  Soundness  of  Approach 

2)  Understanding  of  the  Problem 

3)  Compliance  with  Requirements 


4)  Past  Performance. 

^  Each  "area,  item  and  factor"  of  a  contractor's  proposal  is 
graded  using  the  following  color  code  scheme: 

(a)  Blue:  Exceptional..  Exceeds  specified  performance 
or  capability  in  a  beneficial  way  to  the  Air  Force;  and 
has  high  probability  of  satisfying  the  requirements;  and 
has  no  significant  weakness; 
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(b)  Green;  Acceptable.  Meets  standards;  and  has  good 
probability  of  satisfying  the  requirement;  and  any 
weakness  can  be  readily  corrected; 

(c)  Yellow.  Marginal.  Fails  to  meet  evaluation 
standards;  and  has  low  probability  of  satisfying  the 
requirement;  and  has  significant  deficiencies  but 
correctable ;  and 

(d)  Red;  Unacceptable.  Fails  to  meet  minimum 
requirement;  and  deficiency  requires  a  major  revision  to 
the  proposal  to  make  it  correct. 

Figure  5,  below,  is  an  example  of  how  this  color  scheme  is 
employed. 
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FIGURE-  5;  Color  Rating'  Schema- 


As  will  be  explained  in  more,  detail,  in  subsequent  sections, 
the  assignment  of  color  codes  corresponds'-  directly  to  describing 
the  outcomes  or  values  o£  the  attributes  for  each  decision 
alternative  in  the  context  of.  decision,  analysis;.  For  example,  the 
“Technical  Area,"  might  include  the  factor  “System  Engineering"  and 
the_  item  “Armament  Integration."’  Armamen't  Integration  might  be 
assigned  a  color  "Green"  and  "Low"  risk,  implying  that  the 
alternative  (the  particular  proposal  being,  evaluated)  is  likely 
("low"  risk)  to  attain  an  acceptable  ("Green")  level  of  performance 
on  this  attribute  ('* Armament  Integration").. 


2.4  Advantages  of  the  Current  Source  Selection  Decision  Process 

The  current  evaluation  process,  i.e.  the  assignment  of  colors 
and  levels  of  risk  to  areas,  items,  and  factors,  is  designed  to 
ensure  a  thorough  and  complete  evaluation.  First,  this 
coordination  and  review  process  includes  high  level,  experienced 
individuals,  who  probably  were  evaluators  themselves  on  past  source 
selections.  The  source  selection  decision  is  reviewed  by  several 
"independent  agencies  and  individuals,  from  Headquarters  Air  Force 
Staff  to  the  Product  Division  of  Air  Force  Systems  Command  to  which 
the  SPG  belongs.  Second,  feedback  during  the  entire  source 

selection  process  between  'the  government  and  each  contractor 
promotes  communication,  clarification  of  misunderstandings,  and 
correction  of  discrepancies.  The  government  also  may  solicit 
additional  information  or  ask  for.  clarification  during  Bidder's 
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Conferences  before  the  formal  portion  of  the  source  selection 
begins.  Once  the  formal  source  selection  begins  (i.e.  after  the 
government  receives  the  contractor  proposals) ,  the  communication 
continues  through  contractor  inquiries  and  deficiency  reports. 


2.5  Areas  of  Potential  improvement 

Standardization  among  all  source  selections  is  insured  by  the 
regulations,  directives,  and  policies  of  the  Department  of  Defense, 
the  Air  Force,  Systems  Command  and  the  individual  Product 
Divisions.  The  "envelope”  of  operation  is  narrow  enough  to  protect 
both  the  government  and  contractors  yet  flexible  enough  to  promote 
creativity  and  innovation.  Indeed,  the  Program  Manager  is 
encouraged  to  establish  a  flexible  acquisition  strategy. 

The  one  area,  however,  most  amenable  to  improvement  is  the 
evaluation  of  proposals  and  synthesis  of  these  judgements  to  yield 
a  ranking  of  the  proposals.  It  is  at  this  point  in  the  source 
selection  process  the  evaluators  and  decision  makers  could  most 
benefit  from  decision  analysis. 

Numerical  scoring,  rather  than  the  color  rating  scheme,  was 
used  previously  but  this  method  was  de-emphasized  with  the 
introduction  of  AFR  70-15  in  1984  (Malishenko  (1987)  and  ASD 
Pamphlet  (1981)).  The  current  policy,  as  stated  in  AFR  70- 
15/AAFARS  Appendix  AA,  paragraph  3-4,  is: 

b.  Colors,  symbols,  or  numbers  may  be  used  to  indicate 
proposal  ratings  at  the  factor  and  subfactor  level.  At 
the  item  level,  color  codes  shall  be  used  to  depict  the 
rating  that  has  resulted  from  applying  the  assessment 
criteria  to  the  specific  criteria  in  a  matrix  fashion. 

If  the  SSP  requires  £  summary  rating  at  the  area  level, 
color  codes  shall  be  used.  Ratings  must  be  accompanied 
by  a  consistent  narrative  assessment  of  the  basis  for  the 
rating. 

c.  If  at  any  level  of  indentation,  a  contractor's 
proposal  is  evaluated  as  unacceptable  (fails  to  meet 
minimum  requirements  and  deficiency  requires  major 
revision  to  the  proposal  to  make?  it  correct,  e.g.  color 
code  is  red) ,  this  fact  must  be  included  in  the  rating 
and  narrative  assessment  at  that  element  level  and  each 
higher  element  of  indentation.  Therefore,  a  red  or 
unacceptable  rating  at  any  level  must  be  carried  to  the 
highest  level. 

d.  The  following  elements  are  not  rated; 

(1)  financial  capability.  Production  Readiness 
Reviews,  and  preaward  surveys,  although  these  may 
be  considered  by  the  SSAC. 
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(2)  Cost  (OE'  price) ,  although  these;  may  be  weighted 
by  the  SSAC. 

e.  Color  codes  are-  mandatory  at:  the  item  leveiv  and  if. 
areas  are  rated,  color'  codes  shall  also,  be  used  for.  the 
areas.  There  are  alternative  methods-  that  may  be-  used,  at 
the  factor  and  subfactor  level  to-  accomplish  the>  rating. 

(p.  15) 

In  our  view,  the  current  evaluation  process  suffers  from  the 
following  deficiencies; 

1.  It  is  inconsistent,  in  that  colors,  numbers  and/or 
symbols  can  be  used‘  to  in  the  same  evaluation. 

2 .  There  is  no  theoretically  sound;  way  to  synthesize  the 
many  judgements  into  a  consistent,  logical,  and 
defensible  conclusion-  using  the  current  procedures. 

3.  Color  ratings-  do  not  perniit  enough-  refinement  for  the 
required  judgements. 

4.  The  current  process  is  consistent  with  accepted  or 
tradi-tional  decision  analysis-  until  the  evaluation  and 
synthesis  phases,  " It ^s  closer  but..." 

5.  Areas,  items,  factors,.,  and  subf actors- defined  in  the 
Source  Selectibn  Plan:  precisely  define  a;  -hierarchical 
model  of  the^  source  selection  decision  problem.  Such 
models  are  common,  wall  Jcnowh-,  .  and  have  beon  thoroughly 
studied  in  decision  analysis. 

6.  The  areas,  items,  factors,  and  subfactors  important 
to  the  decision  are,  at  best,  listed  only  by  o.rder  of 
importance.  Measures  of  the  relative  importance  among 
them  are  not  provided. 

7.  Financial  considerations  can  easily  be  incorporated 
as  important  areas,  items,  factors,  or  subfactors  in  the 
decision  model.  If  they  are  considered  by  the  SSAC,  they 
should  be  included  in  the  model. 

Next,  we  present  a  case  study  to  which  decision  analysis  will 
be  applied  to  test  its  usefulness,  efficacy,  and  benefit. 


2 . 6  Case  Study 

Before  introducing  decision  analysis  and  applying  it  to  source 
selection,  we  describe  an  actual  source  selection  currently  in 
progress.  The  sensitive  nature  of  source  selection  precludes  us 
from  identifying  the  particular  weapons  system  and  contractors 
involved.  We  do  describe,  however,  the  actual  criteria  which  were 
used  in  the  evaluation  of  the  proposals.  The  particular 
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evaluations  provided  for  the  case  are  fictitious,  again  with  the 
intent  of  protecting  confidentiality. 

This  source  selection  involves  four  potential  contractors. 
Four  areas  will  be  evaluated.  These  are  listed  below  in  descending 
order  of  importance  along.  Additionally,  each  area  is  further 
broken  down  into  its  constituent  items. 

(1)  Technical 

-  Reliability /Maintainability 

-  Producibility/Quality 

-  Systems  Engineering 

-  Flight  Systems 

-  Avionics 

-  support  Systems 

(2)  Supportability 

-  Organizational  Structure 

-  Management  Policies,  Practices  and  Planning 

-  Supportability  Program  Implementation 

-  Logistics  Capabilities 

(3)  Management  Capability 

-  Program  Management 

-  Future  Competition 

-  Systems  Test 

-  Manufacturing 

-  Quality  Assurance 

-  System  Safety 

-  Training  Systems  Management 

-  Security 

(4)  Cost 

-  Realism 

-  Reasonableness 

-  Completeness 

Each  item  is  further  broken  down  into  factors,  and  factors 
further  broken  down  into  subfactors.  The  complete  listing  of 
areas,  items,  and  factors  for  this  case  is  provided  in  Appendix  1, 
where  they  appear  in  an  indented  paragraph  format. 

Each  of  the  areas,  items,  and  factors  is  rated  against  the 
following  criteria; 

(1)  Soundness  of  Approach 

(2)  Understanding  of  the  Problem 

(3)  Compliance  with  Requirements 

(4)  Past  Performance 

Each  of  the  criteria  are  assumed  to  be  of  equal  importance. 
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3.  DECISION  analysis:  (DA) 

As  the  complexity  in  a  decision  problem:  increases,  the 
objective  data  available;  decreases.,  thereby  forcings  the  decision 
maker  to  rely  more  on  subjective  judgements.  In  these  instances, 
the  literature  shows  that  making  decisions  by  a-  holistic  or 
intuitive  approach  rarely  suffices.  A  decision  maker  cannot 
mentally  assimilate,  simultaneously  consider  and,  cognitively 
process,  all  the  information  involved  in  a  complicated  decision 
problem.  Decision  analysis  (DA)  is  intended  to  assist  the  decision 
maker;  it  is  intended  to  promote  logical  consistent  and  rational 
analysis  to  yield  ”good*'  decisions.”  To  quote,  Keeney  (1982): 

Decision  analyses  is  the  formalization  of  common  sense 

for  decision  problems  which  are  too  complex  for  informal 

use  of  common  sensa>.  (p,  806) 

In  this  paper,  we  consider  tow  decision  analysis  methodologies 
within  the  purview,  of  DA:  multiattribute  utility  theory  (MAUT)  and 
the  analytic,  hierarchy  process  (AHP) . 

Decision  analysis  can  be  either  descriptive  or  normative. 
Descriptive  decision  analysis  is  the  way  decision  makers  actually 
make  decisions,  whereas  normative,  decision  analysis  is  the  way 
decision  makers  should  make  decisions’,  according  to  some  prescribed 
theory.  The  descriptive  research  (Kahneman,  Slovic  &  Tversky, 
1982,  and  von  Winterfeldt  and  Edwards',  1986)  shows  decision  makers 
almost  invariably  violate  the  normative  approaches  to  decision 
making.  These  two  fields  of  research,  are  not  necessarily  distinct. 
A  normative  theory  should  be  based  upon  an  axiomatic  foundation 
that  is  congruent  with  the  way  humans  actually  think  about 
decisions.  The  objective  of  normative  decision  analysis  is,  as 
Keeney  (1989)  states,  to  help  decision  makers  make  more  informed 
and  better  decisions. 

The  major  players  during  a  DA  are  the  decision  maker (s)  (and 
delegated  experts)  and  decision  analyst (s) .  The  decision  maker  is 
the  expert  in  the  decision  domain,  whereas  the  decision  analyst  is 
an  elicitor,  formulator,  and  evaluator  of  information  and 
preferences;  he  is  not  necessarily  an  expert  in  the  problem  domain, 
although  he  does  become  very  familiar  with  it  through  interaction 
with  the  experts  and  decision  makers. 

There  are  several  popular  misconceptions  about  DA.  Decision 
analysis  is  not  a  replacement  for  the  decision  maker,  simply  a 
"decision  tree,"  a  computerized  "black  box,"  or  a  rigid  approach 
that  provides  the  de  facto  solution  to  the  decision  problem. 
Instead,  decision  ils  is  a  process  with  the  purpose  of  helping 
the  decision  maker  make  "better"  decisions.  Decision  analysis 
provides  the  necessary  framework  for  making  consistent,  logical, 
communicable  and  defensible  decisions.  Decision  analysis  focuses 
the  attention  of  the  decision  maker  on  the  important,  as  determined 
by  the  decision  maker,  aspects  of  the  problem.  Howard  (1988) 
summarizes  DA  as: 
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A  systematic  procedure  for  transforming  opaque  decision  problems 
into  transparent  decision  problems  by  a  sequence  of  transparent 
steps.  Opaque  means  ’'hard  to  understand,  solve,  or  explain;  not 
simple,  clear,  or  lucid."  Transparent  means  "readily  understood, 
clear,  obvious."  In  other  words,  decision  analysis  offers  the 
possibility  to  a  decision  maker  of  replacing  confusion  by  clear 
insight  into  a  desired  course  of  action,  (p.  680) 

Decision  analysis  employs  an  analytic  approach,  intended  to 
"divide  and  conquer"  a  complex  problem.  It  first  decomposes  the 
decision  problem  into  its  component  parts: 

1.  alternatives:  choices  or  possible  courses  of  action 

2.  outcomes:  possible  consequences  for  a  particular  choice 

3.  likelihoods  of  outcomes:  possibility  that  an  outcome 
will  occur. 

This  decomposition  allows  the  decision  to  be  broken  into 
manageable  pieces  to  facilitate  understanding,  insight  and  focused 
analysis.  When  a  complex  problem  is  sufficiently  decomposed  and 
defined  as  to  promote  clear  understanding  and  communication,  we 
will  say  it  has  passed  the  clarity  test.  Howard  (1989)  uses  this 
term  also,  but  in  a  more  restricted  sense.  We  will  use  the  term  to 
mean: 


clearly  defining  all  the  decision  problem  components, 
characteristics  and  relevant  information  so  the  decision 
maker  and  decision  analyst  have  a  common  understanding  of 
the  problem  at  hand. 

Often  an  unsuccessful  DA  can  be  ultimately  traced  to  failure 
of  the  clarity  test.  Once  the  clarity  test  is  passed,  the  decision 
analyst  elicits  information  and  preferences  from  the  decision 
maker.  A  structured  model  of  the  decision  problem  is  constructed 
next.  Then,  the  decision  analyst  aggregates  or  synthesizes  the 
judgements  and  information  provided  by  the  decision  maker  into  a 
logical,  consistent,  rational  and  communicable  conclusion. 

Decision  analysis  is  an  iterative  and  interactive  process  (see 
Figure  6)  encompassing  four  basic  steps: 

1.  formulate 

2 .  evaluate 

3.  refine 

4 .  act 

Each  of  these  steps  requires  the  decision  maker's  "blessing" 
before  the  DA  proceeds  to  the  next  step.  The  interaction  between 
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Each  of  .these  steps  requires  the  -decision  mak'eif's  "blessing" 
'before  the  DA  proceeds  to  the  next  step.  The  interaction  between 
the  decision  anaLyst  .and,  decision,  maker  .is  central  to  a  successful 
and  -'credible  DA. 


li 

Insight 

ll 

Action 


FIGURE  6:  The  Decision  Analysis  Process 


Figure  7  depicts  these  four  basic  steps.  The  "formulate"  step 
is  the  foundation  for  the  rest  of  the  analysis — without  a  clear 
understanding  and  representation  of  the  problem,  the  analysis  is 
flawed  from  the  start,  and,  in  essence,  a  "solution"  for  the  "wrong 
problem"  is  the  result. 
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Action 


FIGURE  7:  Relationship  Between  the  Steps  of  Decision  Analysis 


Formulation  involves  identifying  the  problem,  decomposing  the 
problem  into  its  constituent  components,  gathering  information, 
clarity  testing,  and  specifying  model  assumptions.  During  this 
step,  the  decision  analyst  elicits  information  and  judgements  from 
the  decision  maker.  This  interaction  helps  the  decision  analyst 
construct  a  model,  or  representation,  of  the  decision  problem.  The 
level  of  detail  included  in  and  the  complexity  of  the  model  depends 
upon  the  decision  and  decision  maker;  however,  keeping  the  model  as 
simple  as  possible  while  still  capturing  the  espantial 
characteristics  of  the  decision  problem  is  the  objective.  Once  the 
model  is  constructed,  the  decision  maker  and  the  decision  analyst 
discuss  the  model  and  it's  implications.  This  formulation  or  model 
building  step  is  the  most  time  consuming  step  (as  shown  in  Figure 
7)  .  Once  the  decision  maker  agrees  with  the  representation  of  the 
problem,  the  analysis  progresses  to  the  next  step:  evaluation. 

Evaluation  is  the  step  wherein  the  preferred  alternative  is 
determined  according  to  the  theory  and  methodology  of  the 
particular  decision  analysis  technique  being  employed.  For  MAUT, 
the  alternative  yielding  the  "highest  expected  utility"  is  the 
preferred  one  (von  Neumann  and  Morgenstern,  1947)  .  For  the  AHP, 
the  preferred  alternative  is  the  one  with  the  "highest  composite 
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priority,  importance,  or  weighting”  {Barker,  1989). i  The  reader 
nterested  in  the  axiomatic  foundation  and  theory  of  MAUT  and  the 
AHP  are  referred  to  Keeney  and  Raiffa  (1976)  and  ini  Saaty  (1986, 
1988)  respectively.  At  a  minimiam,  the  evaluation  step  will  include 
a  decision  recommendation  and  sensitivity  analysis. 

The  next  step  is  the  refinement  of  the  model.  In  this  step, 
the  analyst  reviews  the  results  with  the  decision  maker  for 
accuracy  of  assessments  and  reasonableness  of  thh  conclusions. 
Discrepancies  help  to  isolate  areas  of  the  analysis  requiring 
further  atterition  and  revision,  thus  the  iterative  nature  of  DA. 
If  the  decision  maker  is  not  satisfied  with  the  analysis  or  if  new 
information  is  available,  the  decision  maker  and  analyst  will 
iterate  back  through  any  or  all  of  the  previous  steps.  This 
refinement  step  allows  the  decision  maker  flexibility  and  promotes 
a  better  understanding  and  further  insight  into  the  problem.  The 
final  step,  action,  is  self-explanatory. 

The  following  quote  from  Howard  (1980)  summarizes  the  essence 
of  decision  analysis: 

Decision  analysis  serves  as  vehicle  for  focusing  all  the 
information  of  experts  that  the  decision  maker  may  wish 
to  bring  to  bear  on  the  problem  while  leaving  the 
decision  maker  free  to  accept,  reject,  or  modify  any  of 
this  information  and  to  establish  preference,  (p.  7) 

The  next  sections  provide  an  overview  of  MAUT  and  the  AHP, 
illustrate  the  application  of  MAUT  and  the  AHP  to  the  case  study, 
and  discuss  advantages  and  disadvantages  of  MAUT  and  the  AHP. 


1  The  concepts  of  "expected  utility"  and  "composite  priority, 
importance,  or  weighting"  will  be  discussed  subsequently  in 
sections  4.2  and  5.2. 
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4.  APPLYING  MAUT  TO  THE  SOURCE  SELECTION  PROBLEM 
4.1  Overview  of  MAUT 

Most  decisions  involve  uncertainty,  risks,  and  conflicting 
objectives.  MAUT  allows  for  the  modeling  of  such  decisions  in  a 
logical,  communicable  and  rational  manner,  consistent  with  the 
axioms  of  utility  theory  as  defined  by  von  Neumann  and  Morgenstern 
(1947) .  Uncertainty  and  risk  are  inherent  in  source  selection. 
Likewise,  source  selection  includes  conflicting  objectives,  e.g. 
maximizing  performance  and  minimizing  cost. 

Even  though  most  decisions  are  complex,  involving 
uncertainties,  risks,  and  conflicting  objectives,  MAUT  can  be  used 
to  capture  the  decision  maker's  preferences,  organize  and 
systematically  incorporate  information  relevant  to  the  particular 
decision  problem.  This  requires  interaction  between  the  decision 
maker  and  the  decision  analyst.  Uncertainties  are  modeled  by  the 
use  of  probability  theory.  Probability  is  the  state  of  knowledge 
or  belief  concerning  an  unknown  future  outcome.  Risk  is  handled 
through  the  use  of  utility  theory.  Utility  is  a  measure  of  the 
value  the  decision  maker  attaches  to  a  particular  outcome.  Utility 
theory  requires  the  decision  maker  to  answer  a  series  of  gamble  or 
lottery  type  questions  of  the  form: 

"Which  do  you  prefer  a  sure  $1000  or  the  gamble:  a  50-50 
chance  of  winning  $4000  or  losing  $2000? 

The  decision  maker's  responses  to  these  questions  provide 
insight  into  his/her  attitude  toward  risk,  as  embodied  in  the 
decision  problem  at  hand.  This  is  because  the  lottery  questions 
are  constructed  using  the  profits  and  costs  relevant  to  the  problem 
of  interest  to  the  decision  maker.  Conflicting  objectives  require 
the  decision  maker  to  make  tradeoffs.  Once  the  uncertainties,  risk, 
and  objectives  are  modeled,  synthesis  is  accomplished  by  computing 
the  weighted  average  of  the  probabilities  and  utilities,  called 
expected  utility. 

The  "best"  course  of  action  is  the  alternative  with  the 
highest  expected  utility.  Maximizing  expected  utility  requires  the 
decision  maker  to  behave  in  the  (normative)  manner  dictated  by  the 
underlying  theory  of  MAUT.  The  decision  maker  must  follow  the 
axioms  of  utility  theory  (Savage,  1954)  or  what  Howard  (1988)  calls 
the  "rules  of  actional  thought."  These  intuitively  appealing 
"rules"  are  descriptive  of  actual  decision  making.  They  are: 

Rule  1:  All  outcomes  can  be  ordered  by  the  decision 
maker  according  to  his  intrinsic  preferences. 

Rule  2:  Given  the  choice  between  a  gamble  of  the  worst 
and  best  outcome,  and  an  certain  outcome,  the  decision 
maker  can  adjust  the  likelihoods  of  the  best  and  worst 
outcomes  until  he  is  indifferent  between  the  gamble  and 
the  certain  outcome. 
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Rule  3.:  The*  decision'  maker  can  substitute  the  g^amble  for 
the  certain  outcome  without  changing-  his  preference 
structure . 

Rule  4;  Given  two  alternatives.,  the  decision,  maker  would 
select  the  alternative,  which  yields;  the  highest  chance,  of 
receiving  the  best  outcome. 

Rule  5:  Independence  of;  uncertainties  is  handled  through 
probability  theory. 

Keeney  (1977)  provides  a*  very  readable  account  of  how  one  puts 
multiattribute  utility  theory  into  practice.  Keeney's  paper 
includes  a  typical  dialogue-  between  the  decision  analyst  (Keeney) 
and  the  decision  maker.  Typical  implementation  of  MAUT  is 
accomplished  in  three  phases;  model  formulation,  evaluation  of  the 
model,  consistency  checking  and  modification. 

Model  formulation  includes;  generating,  alternatives;  clearly 
defining  attributes  and  their  outcomes,  and  likelihood  of  those 
outcomes;  and  determining-  the  utility  function.  Determining  the 
utility  function  is  time  consuming  and  requires  an  experienced  and 
trained  decision  analyst.  In  particular,  the  decision  analyst  must 
verify  independence  assumptions,  assess,  indifference  tradeoffs 
among  attributes,  and  assess  the  decision  maker's  utility  functions 
for  each  of  the  attributes. 

The  second  step  is  evaluating  the-  model  using  expected  utility 
theory.  Also,  sensitivity  analysis  is  conducted  to  provide 
additional  insight  into  the  decision  problem. 

The  final  phade  provides  the  decision  maker  with  consistency 
checks,  and  the  model  is  modified  if  a,  consistency  check  fails. 

Corner  and  Kirkwood  (1988)  provide  a  bibliography  of  MAUT 
applications.  Additionally,  Goren  (1981)  discusses  the  potential 
for  applying  Edwards'  (1977)  Simple  Multiattribute  Rating  Technique 
(SMART) ,  a  simplified  version  of  MAUT  as  presented  in  Keeney  and 
Raiffa  (1976) ,  to  source  selection.  MAUT  has  been  successfully 
applied  in  a  wide  range  of  areas  including  energy,  manufacturing, 
medicine,  services,  and  public  policy.  Applications  of  MAUT  have 
proven  effective  in  contractor  selection  for  energy  supplies, 
evaluating  solar  energy  project  proposals,  and  the  determination  of 
development  strategies  for  542  proposed  Norwegian  hydroelectric 
projects.  These  applications  illustrate  the  ability  of  MAUT  to 
assist  in  the  solution  of  decision  problems  with  characteristics 
similar  to  those  of  the  source  selection  problem. 
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4.2  Applying  MAUT  to  the  source  Selection  Case  Study 

The  application  of  MAUT  to  problems  of  similar  scope  and 
structure  as  source  selection  and  the  valuable  insight  that 
resulted  from  each  provide  ample  support  for  the  use  of  MAUT  in 
source  selection  decisions.  However,  applying  MAUT  to  the  source 
selection  problem  requires  the  development  of  a  different  protocol 
than  is  presently  used  in  source  selection. 

The  decision  maker  must  be  identified,  who  then  delegates 
experts  to  act  in  his  behalf.  Each  expert  would  represent  the 
various  functional  and  technical  disciplines  relevant  to  the 
particular  acquisition.  These  experts  would  familiarize  themselves 
with  the  standards  and  contract  requirements  of  the  acquisition 
project.  This  effectively  creates  the  foundation  from  which  the 
model  will  be  developed.  The  model  should  capture  the  decision 
maker's  (expert's)  knowledge,  information  and  preferences 
concerning  the  source  selection.  Therefore,  using  MAUT  in  Air 
Force  source  selections  requires  a  departure  from  the  current 
process.  However,  MAUT  offers  a  logical,  defensible,  and 
consistent  approach  to  the  source  selection  decision  process.  The 
next  few  paragraphs  illustrate  how  MAUT  could  be  applied  to  the 
source  selection  case  study  described  in  Section  2. 

We  applied  MAUT  to  the  source  selection  case  study  following 
the  three  phases  outlined  in  4.1.  The  first  phase  calls  for  the 
formulation  of  the  model.  This  requires  the  generation  of  the 
possible  decision  alternatives,  a  step  that  is  easily  accomplished 
in  a  source  selection  decision  problem.  Alternatives  are  simply 
contractor  proposals.  For  the  case  study,  there  were  four 
contractors.  This  phase  also  requires  the  determination  of 
attributes,  which  dictates  the  structure  of  the  model.  These 
attributes  are  determined  by  interaction  between  the  decision 
analyst  and  the  decision  maker /experts.  For  a  successful  decision 
analysis,  attributes  must  be  clearly  defined  and  understood.  If 
necessary,  this  will  require  the  decision  maker  to  decompose  or 
refine  the  attribute  into  lower  levels,  i.e.,  items,  factors,  and 
subfactors.  The  level  of  detail  required  is  determined  through 
interaction  with  the  decision  analyst  and  the  decision  maker.  The 
decision  analyst  clarifies  the  attributes  which  must  be  examined  by 
helping  to  focus  the  expert's  attention  and  thought.  The  detail 
required  to  clearly  define  each  attribute  is  problem  specific  and 
depends  on  the  importance  as  well  as  the  scope  of  that  attribute. 
An  attribute  should  only  be  broken  down  to  the  level  at  which  the 
experts  can  make  confident  judgements. 

For  the  case  study,  it  was  determined  that  four  major 
attributes  that  were  important  in  evaluating  the  decision.  These 
attributes  are:  technical  success  of  the  weapon  system, 
supportability  of  the  weapon  system  (including  logistic 
considerations) ,  management  capability  of  the  contractor,  and  cost. 
Each  attribute  was  broken  down  to  the  item  level  as  outlined  in 
Appendix  1. 
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Next,  the  possible  outcomes  of  the  source  selection  decision 
are  determined.  For  the  case  study, the  outcomes  were  the  levels  of 
performance  or  capability  a  proposal  could  achieve.  These  levels 
of  success  are  the  colors  currently  used  in  source  selection: 
blue,  green,  yellow,  and  red,  corresponding  to  exceptional, 
acceptable,  marginal,  and  unacceptable  performance. 

The  evaluation  t-eams  would  next  use  expert  judgement  to  assess 
the  likelihoods  of  each  contractor  attaining  these  outcomes  on  each 
of  the  "items.”  The  decision  analyst  elicits  this-  knowledge  in 
terms  of  probabilities.  Spetzler  and  Stael  von  Holstein  (1975) 
discuss  techniques  in  assessing  probabilities  for  uncertain 
outcomes.  For  the  case  study,  the  probability  of  technical  success 
was  influenced  by  reliability  and  maintainability  of  the  system, 
producibility  and  quality  of  engineering,  system  engineering, 
flight  systems,  avionics,  and  support  systems.  Appendix  2  contains 
the  assumed  probabilities  associated  with  the  possible  outcomes. 
These  probabilities  were  determined  at  the  item  level  (see  Appendix 
1). 


The  next  step  in  formulating  the  model  requires  the  decision 
analyst  to  determine  the  decision  maker's  intrinsic  preferences 
structure  for  the  various  outcomes.  He/ she  does  this  by  eliciting 
"utility  functions."  In  this  step  the  decision  analyst  verifies 
independence  assumptions,  assesses  indifference  tradeoffs  among 
attributes,  and  assesses  the  decision  maker's  utility  functions  for 
each  of  the  attributes.  Since  these  assessments  are  mostly 
mechanical,  we  provide  only  a  summary  account.  Further  discussion 
regarding  utility  function  assessment  can  be  found  in  Keeney  and 
Raiffa  (1976) .  From  Watson  and  Buede  (1987) ,  preference 
independence  is  defined  as; 

A  pair  of  attributes,  XI,  X2,  is  said  to  be  preference 
independent  of  all  the  other  attributes  {Xi,  i  =  3,...,n} 
if  preferences  between  different  combinations  of  levels 
of  XI,  X2,  with  the  level  of  all  other  attributes  being 
held  at  constant  values,  do  not  depend  on  what  these 
constant  values  are.  (p.  26) 

Also,  the  decision  analyst  verifies  utility  independence. 
Again  from  Watson  and  Buede  (1987) : 

An  attribute  X  is  said  to  be  utility  independent  of  other 
attributes  Y,  Z,...  if  the  decision  maker's  preference 
amongst  gambles  on  X  do  not  depend  on  the  levels  of  the 
other  attributes,  (p.  64) 

For  the  case  study,  we  assumed  both  preference  and  utility 
independence. 

Next,  the  scaling  constants  (or  importance  weights  of 
attributes)  are  determined.  Normally,  this  requires  the  decision 
maker  to  make  tradeoffs  among  attributes.  For  the  case  study,  we 
assumed  reasonable  scaling  constants.  Once  independence  is  assumed 
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and  scaling  weights  are  assessed,  the  functional  form  of  the 
utility  function  can  be  determined.  How  the  analyst  determines  the 
precise  mathematical  formula  for  the  utility  function  is  beyond  the 
scope  of  this  paper;  the  interested  reader  is  referred  to  Bell 
(1979) .  Edwards  (1977)  suggests  the  additive  utility  function  is  a 
good  approximation  to  many  actual  utility  functions.  We  thus 
assumed,  for  illustrative  purposes,  the  decision  maker  for  this 
case  has  an  additive  utility  function  of  the  form: 

U(t,  m,  s,  c)  =  Kt  U(t)  +  Km  U(m)  +  Ks  U(s)  +  Kc  U(c) , 
where : 

t  is  the  technical  attribute, 
m  is  the  management  attribute, 
s  is  the  supportability  attribute, 
c  is  the  cost  attribute, 

Kt  is  the  importance  weight  for  the  technical  attribute; 
likewise  for  Km,  Ks,  and  Kc,  and 

U(t)  is  the  utility  function  for  the  technical  attribute, 
likewise  for  U(m),  U(s),  and  U(c). 

Sample  calculations  are  contained  in  Appendix  2.  Utility 
functions  for  each  attribute  were  assessed  by  the  method  presented 
in  Keeney  (1977)  and  Farquhar  (1984). 

Next,  the  model  is  "evaluated.”  MAUT  dictates  that  the 
preferred  alternative  is  the  one  with  the  highest  expected  utility. 
This  computation  is  accomplished  by  "rolling  back"  the  decision 
tree  as  outlined  in  Raiffa  (1968). 

Appendix  2  contains  a  MAUT  model  representing  the  case  study 
of  Appendix  1.  The  model  represents  four  alternatives 

(contractors)  and  four  attributes  (technical,  supportability, 
management,  and  cost) .  It  also  contains  two  or  three  outcomes 
(blue  or  green  or  yellow)  for  each  attribute.  The  resultant  model 
is  represented  in  a  large  decision  tree  (122  end  points).  Of  the 
software  packages  ARBORIST  (Samson,  1988) ,  SUPERTREE  (McNamee  and 
Celona,  1987),  and  EXPRESSION  TREE  (Cheung  and  Kirkwood,  1989), 
only  EXPRESSION  TREE  had  the  capability  to  handle  a  model  of  this 
size.  This  package  required  only  that  eighteen  nodes  be  entered 
and  was  easy  to  use. 

The  MAUT  analysis  resulted  in  contractor  C  as  the  preferred 
alternative.  Sensitivity  analysis  of  the  technical  attribute  was 
conducted  by  varying  the  likelihood  of  obtaining  a  "blue"  or 
"green"  outcome.  We  also  tested  the  sensitivity  of  the  decision  to 
the  scaling  constants  for  the  utility  functions  (Kt,  Ks,  Km,  and 
Kc)  .  Contractor  C  remained  the  preferred  choice  over  all  ranges 
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for  these  parameters.  Such  an  analysis  would  ;ijpt  have  been 
possible  without  a  software  package  like  EXPRESSION  TREE. 

The  last  phase  when  applying  MAUT  to  source;  selection  is 
checking  for  consistency  in  the  model  and  reasonableness  of  the 
utility  function  assessments,  and,  if  necessary,  revision  and 
reassessment.  Typically,  the  decision  maker  would  review  the  model 
and  preliminary  results  to  ensure  that  the  utility  function 
accurately  represents  judgments  and  risk  attitude  of  the  decision 
maker.  Also,  the  probabilities  associated  with  the  outcomes  are 
reexamined  and  reassessed  if  ne.cessary. 
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4.3  Advantages  of  MAUT 

There  are  several  advantages  that  MAUT  provides  over  the 
present  decision  method  of  source  selection.  The  biggest 
reservation  about  the  use  of  a  quantitative  methods  in  decision 
analysis  is  doubt  about  the  ability  to  incorporate  the  subjective 
aspects  of  the  decision.  Huber,  et  al  (1969)  have  examined  studies 
that  strongly  indicate  that  professionals  can  develop  and  reliably 
use  subjective  evaluation  models. 

The  insight  provided  by  the  hierarchical  decomposition  of  the 
problem  certainly  can  not  be  overlooked.  The  decision  problem  is 
broken  down  into  manageable  pieces  and  the  important  aspects  of  the 
problem  are  presented  in  a  logical,  consistent  and  rational  manner. 
Much  of  the  benefit  of  using  MAUT  comes  from  the  in-depth  look  at 
the  problem  and  clarification  of  the  important  aspects  of  the 
problem  which  are  essential  for  the  development  of  an  adequate 
model.  In  particular,  communication  is  enhanced  because  of  the 
necessity  to  focus  discussion  on  important  issues  and  to  force  the 
decision  maker  to  organize  and  structure  his  or  her  thinking. 
Overall,  a  greater  understanding  and  familiarity  with  the  decision 
at  hand  is  facilitated  by  model  building.  This  is  an  important 
characteristic  of  this  process. 

Another  important  aspect  of  MAUT  is  the  ability  to  incorporate 
sensitivity  analysis  into  its  'structure.  The  probabilities, 
scaling  constants,  and  utilities  associated  with  a  model  can  be 
varied  through  a  range  of  possible  values  to  examine  the  effects  on 
the  decision.  Some  areas  may  prove  so  insensitive  that  no  value 
would  change  the  end  result,  whereas  others  may  be  very  sensitive 
to  changes  in  model  parameters.  Thus,  sensitive  areas  can  be 
identified,  effectively  drawing  attention  to  the  critical  aspects 
of  a  particular  acquisition  problem.  This  sensitivity  analysis 
provides  further  insight  into  the  decision  problem. 

One  final  notable  advantage  of  MAUT  over  the  current  source 
selection  decision  process  is  separating  the  analyst  role  from  the 
experts.  The  current  process  requires  the  source  selection 
evaluators  to  be  both  technical  expert  and  decision  analyst.  By 
separating  the  two,  the  expert  can  concentrate  on  knowledge  and 
information  concerning  the  decision  problem,  while  the  analyst 
concentrates  on  modeling  and  analyzing  the  decision  problem  as 
perceived  by  the  decision  maker(s).  However,  both  must  interact 
and  communicate  clearly  in  order  to  conduct  an  effective  decision 
analysis.  Separating  the  experts  and  analysts  would  most  likely 
provide  a  more  independent,  unbiased,  and  objective  source 
selection  evaluation.  In  the  course  of  asking  questions  to  develop 
the  model,  the  decision  maker  could  be  presented  with  a  new  angle 
or  problem  characteristic  not  previously  considered.  In  summary, 
the  use  of  an  independent  decision  analyst  would  serve  as  a  check 
on  the  source  selection  process  to  ensure  that  the  model  and 
evaluations  are  truly  reflective  of  the  decision  maker's  thinking. 
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4.4  Disadvantages  6f  MAUT 

There  are  rdisadvantages  lassociated  with  incofporating  MAUT 
into  .source  selection.  Howard  (1980)  ?notes  the  ^ality  of  'the 
■analysis  depends  critically  'on  '.the  quality  of  'the  decision  analyst,. 
Model  clarity,  ^realism  and  .usefulness  ’are  directly  Impacted  by  the 
analyst's  abilities. 

Because  MAUT  involves  a  'departure  from  the  [current  source 
selection  process,  there  will  undoubtedly  be  resistance  to  change. 
But,  just  because  the  decision  analysis  process  is  involved  and 
different  from  current  methods  does  not  mean  it  shouldn't  be  used. 
Howard  draws  this  analogy  (,1980) ; 

Because  effective  brain  surgery  is  difficult  does  not 
mean  that  there  is  anything  wrong  with  it.  I  would  no 
more  expect  a  person  with  little  training  to  complete  an 
effective  decision  analysis  than  I  would  expect  him  to 
perform  a  successful  brain  operation.  (pp.  14-15) 
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5.  SOURCE  SELECTION  PROCESS  USING  RHP 
5.1  Overview  of  RHP 

The  AHP  is  a  relatively  new  method  developed  by  Professor 
Thomas  L.  Saaty  of  the  University  of  Pennsylvania.  He  began 
development  of  AHP  in  the  early  1970 's  and  the  process  continues  to 
be  refined  and  improved  today.  While  the  AHP  is  not  without 
adamant  opponents  (discussed  below  in  Section  5.5)  there  have  been 
many  applications  to  aid  the  analysis  of  complex  decision  problems 
with  new  applications  appearing  in  the  literature  nearly  every  day. 
Golden,  et  al  (1989),  Saaty  (1988)  and  Zahedi  (1986)  give  an 
extensive  list  of  applications  in  such  diverse  areas  as  conflict 
resolution,  portfolio  selection,  budget  allocation,  accounting  and 
auditing.  There  have  been  several  conferences  and  special  journal 
issues  devoted  exclusively  to  the  AHP. 

At  the  risk  of  oversimplification,  the  AHP  proceeds  basically 
as  follows; 

1.  Analyze  the  components  of  the  decision  problem.  This 
involves  determination  of  the  decision  goal,  criteria, 
sub-criteria,  attributes  and  alternatives. 

2.  Specify  how  the  components  are  related.  Delineation 
of  the  components  and  relationships  results  in  a 
hierarchical  model  of  the  decision  problem. 

3.  For  criteria,  sub-criteria,  and  attributes  at  the 
same  level  of  the  hierarchy,  determine  the  relative 
importance,  priority  or  weighting  of  the  corresponding 
criteria  or  sub-criteria.  This  is  accomplished  by  the 
decision  maker  judging,  usually  in  a  pairwise  fashion, 
which  criterion  is  more  "important”  than  the  other  and 
specifying  how  much  more  important  on  a  nine  point  scale. 

This  step  will  result  in  a  priority  of  criteria,  sub¬ 
criteria  and  attributes  with  each  receiving  a  numerical 
rating.  The  priorities  of  criteria,  sub-criteria  and 
attributes  of  the  same  level  of  the  hierarchy  are 
positive  and  sum  to  one.  A  "consistency  ratio"  is 
calculated  to  assess  how  consistent  the  decision  maker 
was  in  his  pairwise  comparisons. 

4.  Alternatives  are  then  compared  by  the  decision  maker, 
again  in  a  pairwise  fashion,  with  respect  to  each 
criteria,  sub-criteria  or  attribute  at  the  level  of  the 
hierarchy  just  above  the  alternatives.  He  or  she  judges 
which  of  the  two  alternatives  measures  higher  on  the 
associated  attribute,  and  specifies,  again  on  a  nine 
point  scale,  by  how  much  higher.  These  assessments  are 
conducted  either  verbally,  graphically,  or  numerically. 

This  step  results  in  a  relative  score  for  each 
alternative  with  respect  to  each  criteria,  sub-criteria, 
or  attribute,  with  scores  being  positive  and  summing  to 
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one,  A  measure  of  consistency  is  again  calculated  for 
these  comparisons. 

5.  Judgements  are  synthesized  to  determine  ,an  overall 
composite  priority  for  each  alternative.  The  alternative 
with  the  highest  composite  priority  is  the  preferred 
alternative. 


5.2  An  Example  of  the  AHP 

An  example,  taken  from  Saaty  (1988),  will  serve  to  illustrate  the 
fundamentals  of  the  AHP.  The  goal  of  the  decision  problem  is  to 
select  the  "best"  school  to  attend  from  among  three  alternatives; 
A,  B,  or  C.  The  decision  is  to  be  based  upon  six  criteria; 
learning  (L) ,  friends  (F) ,  school  life  (S) ,  vocational  training 
(V) ,  college  preparation  (C)  and  music  classes  (M) .  After 
completing  steps  1  and  2,  the  hierarchical  model  is  depicted  below; 


Satisfaction  with 
school 


Step  3  of  AHP  proceeds  with  the  decision  maker  comparing  the 
six  criteria  in  a  pairwise  fashion  to  determine  the  relative 
importance  of  each.  He  or  she  would  be  asked  a  series  of  questions 
like  the  following; 

“With  respect  to  the  overall  goal  of  satisfaction  with 
the  school  chosen,  which  is  most  important  in  the 
decision;  Learning  or  Friends?” 

Assuming  the  decision  maker  indicated  that  Learning  was  more 
important,  he  would  then  be  asked; 
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"Indicate  how  much  more  important  Learning  is  than 
Friends  on  the  scale  below: 

1.  Equally  important 

2. 

3.  Weak  importance  of  one  over  the  other. 

4. 

5.  Essential  or  strong  importance. 

6. 

7.  Very  strong  or  demonstrated  importance. 

8. 

9.  Absolute  importance." 

(NOTE:  2,  4,  6,  and  8  are  simply  intermediate  values  between 

adjacent  scale  values.) 

Pairwise  comparisons  would  be  made  for  all  possible  pairs  of 
criteria  and  the  results  summarized  in  a  "comparison  matrix,"  like 
the  one  below.  The  entry  in  the  ith  row  and  jth  column  is  the 
relative  importance  of  criterion  i  with  respect  to  criterion  j 
according  to  the  above  scale.  Notice  that  the  matrix  is  a 
reciprocal  matrix  with  the  entry  in  the  ith  row  and  jth  column 
being  the  inverse  of  the  entry  in  the  jth  row  and  ith  column'. 


L 

F 

S 

V 

C 

M 

L 

1 

4 

3 

1 

3 

4 

F 

1/4 

1 

7 

3 

1/5 

1 

S 

1/3 

1/7 

1 

1/5 

1/5 

1/6 

V 

1 

1/3 

5 

1 

1 

1/3 

C 

1/3 

5 

5 

1 

1 

3 

M 

1/4 

1 

6 

3 

1/3 

1 

TABLE  i:  Comparison  Matrix  for  Criteria 

The  relative  weighting  or  importance  of  for  each  criteria  is 
determined  from  the  pairwise  matrix. 2  in  this  instance,  the  weights 
of  the  criteria  are: 


_L _ F  S  V  C  M 

0.32  0.14  0.03  0.13  0.24  0.14 


2  These  weights  or  importance  values  are  obtained  from  the 
normalized  eigenvectors  of  the  pairwise  comparison  matrix  which 
corresponds  to  the  maximum  eigenvalue.  The  interested  reader  is 
referred  to  Saaty,  (1988) . 
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TABLC  2 :  Relative  Weights  of  the  CriteMa 


A  measure  of  consistency,  called  the  consisteritsy  ratio  (C.R.) 
is  then  calculated. ^  A  C.R.  of,  less  than  0.10  is  considered  by 
Saaty  to  be  acceptable,  whereas  a  C.R.  above  O.lO  would  suggest 
that  the  comparisons  be  reaccompHshed.  The  C.R.  for  the  above 
comparison  matrix  is  a  relatively  inconsistent  0.24,  implying  this 
decision  maker  should  be  asked  to  reaccomplish  these  pairwise 
comparisons 

Step  4  compares,  again  in  a  pairwise  fashion,  the  alternatives 
with  respect  to  each  of  the  criteria.  These  comparisons  are 
summarized  in  the  reciprocal  matrices  below: 


With  respect  to: 
Learning 

_A _ B  , _ C 


B 

C 


1  1/3  1/2 

3  13 

2  1/3  1 


With  respect  to: 
Friends 


ABC 
A 
B 
C 


111 

111 

111 


With  respect  to: 
School  life 


A 

B 

C 


ABC 
15  1 

1/5  1  1/5 

15  1 


C.R.  =  0.04 


C.R.  «  O.CO 


C.R.  *  0.00 


With  respect  to:  With  respect  to: 

Vocational  trng.  College  prep. 


With  respect  tc: 
Music  classes 


A 

B  C 

A 

B 

c 

A 

B 

C 

A 

1 

9  7 

A 

.1 

1/2 

"i” 

A 

1 

6 

4 

B 

1/9 

1  1/5 

B 

2 

1 

2 

B 

1/6 

1 

1/3 

C 

1/7 

5  1 

C 

1 

1/2 

1 

C 

1/4 

3 

1 

C.R. 

=  0.18 

C.R. 

=  0.00 

C.R. 

=  0.04 

TABLE  3:  Comparison  Matrices  for  Alternatives 
with  Respect  to  Criteria 

The  relative  scores  of  each  alternative  with  respect  to  each 
criteria  are  given  in  the  Table  4  below. 


School 

Learning 

Friends 

School 

life 

Vocational 

training 

College 

prep. 

Music 

classes 

A 

■ElfOBBi 

0.33 

0.45 

0.77 

0.25 

0.69 

B 

0.33 

0.09 

0.05 

0.50 

0.09 

C  1 

0.33 

0.46 

0.17 

0.25 

0.22 

3  The  interested  reader  is  again  referred  to  Saaty  (1988)  for  a 
detailed  discussion  of  "consistency  ratio." 
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TABLE  4:  Relative  Scores  of  the  Alternatives  with 
Respect  to  the  Criteria 


Step  5  combines  the  relative  scores  with  the  weights  of  the 
criteria  in  a  straight  forward  multiplicative  fashion  to  obtain  the 
composite  priority  for  each  aiternative.  The  composite  priority 
for  a  particular  school  is  obtained  by  multiplying  each  score  for 
that  school  (Table  4)  by  the  corresponding,  weighting  for  the 
criterion  (Table  2)  and  summing  the  resultant  products.  For  , 
example,  the  composite  priority  for  school  A  is: 

0.16*0.32  +  0.33*0.14  +  0.45*0.03  +  0.77*0.13  +  0.25*0.24  + 

0.69*0.14  =  0.37 

The  composite  priorities  for  the  alternatives  are: 


School 

Composite 

Priority 

A 

0.37 

B 

0.38 

0.25 

TABLE  5:  Composite  Priorities  of  Alternatives 


Thus,  schools  B  and  A  are  almost  equivalently  preferred  over  C 
by  this  decision  maker. 

There  is  much  more  to  the  AHP  than  has  been  described  here. 
The  above  discussion  is  not  intended  to  be  a  complete  treatise  on 
the  AHP,  but  instead  to  provide  the  reader  with  a  "feel"  for  the 
"mechanics"  of  the  process.  The  reader  is  referred  to  Saaty  (1988) 
for  the  details,  rationale  and  theory  underlying  the  AHP. 


5.3  Applying  the  AHP  to  the  source  Selection  Case  Study 

The  AHP  is  well  suited  for  application  to  the  source  selection 
problem.  The  most  appealing  feature  of  the  AHP  for  this  problem 
domain  is  its  capability  to  incorporate,  in  a  logical,  rational  and 
consistent  way,  the  subjective  judgements  of  the  decision  maker. 
As  Vincze  (1990)  recently  stated  in  his  review  of  the  AHP  as 
embodied  in  the  EXPERl''  CHOICE  (1990)  software  package: 

Initially,  the  feature  that  interested  this  reviewer  is 
that  EXPERT  CHOICE'S  decision  analysis  does  not  require 
numerically  based  judgements.  EC  allows  subjective 
judgement  about  aspects  of  a  problem  for  which  no  scale 
of  measurement  exists.  Again,  quoting  from  the  manual, 

"Your  subjective  judgements  are  applied  in  a  systematic 
way  to  solve  your  problems.  Expert  Choice  does  not  make 
a  choice  for  you  in  some  mysterious  way,  but  helps  you  to 
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make  an  expert  choice  based-  on  your  knowledge, 
experience,  and  preferences.”  (p.  10) 

Other  advantages  that  suggest  the  application  of  AHP  to  the 
source  selection  problem  are  ease  of  use,  readily  available  and 
relatively  inexpensive  sof-ware  (EXPERT  CHOICE  (1^90),  CRITERION 
(1989)),  and  intuitive  appeal.  Most  importantly  though,  the  current 
source  selection  process  might  be  improved  through  a  more 
systematic,  rational  and  consistent  evaluation  of  alternatives  and 
synthesis  of  judgements  into  a  priority  of  alternatives. 

In  the  remainder  of  this  Section  we  describe  the  application 
of  the  AHP  to  the  case  study  introduced  in  Section  2. 

In  Section  5.1,  we  described  the  AHP  and  the  basic  process 
embodied  in  the  following  five  steps; 

1.  Analyze  the  problem  components. 

2.  Specify  component  relationships. 

3.  Determine  relative  importance  of  each  component. 

4.  Compare  alternatives  with  respect  to  each  criteria. 

5.  Determine  each  alternative's  composite  priority. 

It  is  interesting  at  this  point  to  investigate  the 

correspondence  between  the  currant  source  selection  process  and  the 
AHP  as  embodied  in  the  above  five  steps.  The  current  source 
selection  process  clearly  accomplish  steps  1  and  2  by  defining  the 
areas,  items,  factors,  subfactors,  and  assessment  criteria.  In 
fact 

the  areas,  items^  factors,  subfactors,  and  assessment 
criteria  clearly  defines  the  hierarchical  model  of  the 
source  selection  problem  as  viewed  by  the  Source 

Selection  Advisory  Council  and  the  Source  Selection 
Evaluation  Board. 

That  is  to  say,  the  areas,  items,  factors,  subfactors,  and 
assessment  criteria  for  the  case  study,  as  provided  in  Appendix  1, 
is  equivalent  to  the  hierarchical  model  of  Appendix  3. 
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step  3  is  also  addressed  in  the  current  source  selection 
process.  The  Source  Selection  Advisory  Council  lists  the  criteria 
of  importance  in  the  source  selection  in  descending  order  of 
importance.  This  ordinal  ranking  of  criteria  is  even  communicated 
to  the  potential  contractors.  Moreover,  the  System  Program  Office 
often  precisely  and  numerically  specifies  the  relative  importance 
of  the  criteria .  Usually,  however,  these  weights  are  determined  in 
an  ad  hoc  manner.  But, 

the  AHP  provides  a  theoretically  based,  systematic, 
intuitively  appealing  methodology  to  incorporate  and 
summarize  the  decision  maker's  subjective  judgements  into 
the  required  relative  weights. 

The  comments  regarding  Step  3  apply  equally  to  Step  4.  The 
AHP  assists  the  decision  maker  to  consistently,  systematically,  and 
subjectively  judge  how  well  each  alternative  satisfies  the  criteria 
important  for  the  decision  problem  at  hand.  In  the  case  of  source 
selection,  this  evaluation  step  is  performed  by  the  separate  "Item 
Teams”  as  described  in  Section  2. 

There  are  two  ways  of  evaluating  alternatives  in  the  AHP: 

1.  pairwise  comparisons  between  alternatives  using 
relative  measurement, 

2.  evaluation  of  each  alternative  with  respect  to  each 
criterion  using  absolute  measurement. 

For  the  fourth  step,  we  used  absolute  measurement  in  lieu  of 
pairwise  comparisons  as  discussed  above.  The  color  rating  scheme 
is  the  basis  of  the  absolute  measurements  used  in  our  AHP  model. 
For  each  criterion  at  the  lowest  level  of  the  hierarchy,  pairwise 
comparisons  were  initially  employed  to  determine  the  relative 
importance  of  the  colors  with  respect  to  each  of  these  criteria. 
This  permits  item  specific  meanings  for  the  color  ratings.  For 
example,  the  relative  importance  of  ”  red"  and  "blue"  will  most 
likely  be  dramatically  different  when  rating  "System  Safety"  versus 
"Organizational  Structure." 

Another  advantage  of  using  absolute  measurement  is  that 
potential  of  "rank  reversal"  (described  in  Section  5.5)  is  avoided. 

Furthermore,  regulations  governing  the  source  selection 
process  prohibit  comparisons  between  contractors;  instead 
contractor  performance  is  compared  against  standards  established  in 
the  Source  Selection  Plan: 

2 -9a,  The  SSE8  conducts  its  evaluation  by  measuring  each 
proposal  against  objective  standards  established  at  the 
lowest  level  of  subdivision.  The  SSEB  shall  not  compare 
proposals  against  each  other,  (Air  Force  Regulation  70-15 
(1988),  p.  14) 
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A  hierarchical  model  with  the  size  and  complexity  of  the  case 
study  requires  many  pairwise  comparisons.  It  is  possible  however, 
to  reduce  the  number  of  comparisons  required,  but  the  number  and 
time  required  are  still  substantial.  These  time  ^Savings  can  be 
attained,  in  large  part,  by  use  of  absolute  versus  relative 
measurement  (Liberatore  1989) . 

For  these  reasons,  absolute  measurement  is  use'd  in  Step  4  of 
our  application  of  the  AHP  to  the  case  study.  The  alternatives 
(potential  contractors)  were  “ranked  in  reference  to  standards" 
(Saaty  1987) . 

The  AHP  can  perhaps  provide  the  greatest  improvement  to  the 
current  process  with  Step  5.  The  current  process  approaches  bona 
fide  decision  analysis  by  structuring  the  hierarchical  model, 
specifying  the  relative  importance,  at  least  ordinally,  of  the 
criteria,  and  evaluating  alternatives  against  criteria.  But,  once 
evaluations  of  alternatives  are  completed,  the  current  process 
reverts  to  an  ad  hoc,  intuitive  means  of  determining  the  final 
ranking.  The  AHP  on  the  other  hand,  provides  a  systematic, 
logically  defensible,  and  insightful  way  of  synthesizing  these 
evaluations  into  the  corresponding  priority  of  alternatives 
(contractors) . 

In  Step  5,  the  color  ratings,  beginning  with  each  of  the 
lowest  level  criteria,  are  synthesized  into  a  final  priority  of 
contractors.  The  rating  for  each  contractor  on  each  criterion  was 
obtained  independent  of  the  other  contractor's  proposal  and  each 
contractor  was  compared  only  to  standards  established  for  the 
system  acquisition.  The  overall  ranking  of  contractors  is  obtained 
through  the  method  illustrated  in  Section  5.2. 

In  our  application  to  the  case  study,  we  investigated  ways  to 
translate  the  numerical  "composite  priorities"  provided  by  the  AHP 
back  into  a  traditional  color  rating.  This  makes  the  application 
of  AHP  to  source  selection  consistent  with  established  procedures. 
Moreover,  the  AHP  methodology  is  almost  totally  "hidden"  from  the 
users,  giving  the  appearance  that  they  are  "conducting  business  as 
usual."  It  should  be  emphasized  that 

1 .  all  of  the  advantages  of  applying  the  AHP  are 
retained^ 

2 .  source  selection  proceeds  as  specified  in  existing 
regulations^  directives  and  policies,  and 

3 .  evaluators  and  decision  makers  are  not  confronted 
with  changing  to  and  learning  a  dramatically  different 
method. 

In  this  application,  the  "composite  priorities"  (numerical 
ratings)  were  translated  into  the  colors  currently  used  in 
evaluation.  This  was  done  so  the  implementation  of  the  AHP  would 
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be  consistent  with  the  current  procedures  and  appear  familiar  to 
the  evaluators  and  decision  makers.  This  was  done  as  follows: 

1.  For  the  criterion  being  evaluated,  set  all 
evaluations  to  "blue.”  Note  the  resultant  "composite 
priority"  provided  by  the  AHP.  This  provides  the  upper 
limit  for  the  "blue  range." 

2.  Set  all  evaluations  to  "green."  Note  the  resultant 
"composite  priority." 

3.  The  midpoint  or  average  of  the  "composite  priorities" 
obtained  in  steps  1  and  2  is  the  lower  limit  of  the  "blue 
range"  and  the  upper  limit  of  the  "green  range." 

4.  Repeat  steps  1  through  3  for  the  "green"  and 
"yellow. " 

As  an  example,  consider  the  hypothetical  results  below: 


Evaluations 

Resultant 

Color 

all  set  to: 

"Composite  Priority"; 

Average : 

Blue 

0.51 

0.425 

0.425  -  0.510 

Green 

0,34 

0.230 

0.230  -  0.425 

Yellow 

0.12 

0.085 

0.085  -  .230 

Red 

0.05 

TABLE  6:  Translation  of 

Priorities  to 

Colors 

Thus,  a  "Composite  Priority"  of  .474  would  be  translated  to  a 
color  code  of  "blue,"  .383  to  "green,"  .215  to  "yellow."  A  "red" 
evaluation  is  required  by  current  regulations  to  be  carried  up  all 
higher  of  the  hierarchy. 

While  translation  of  the  numerical  ratings  to  colors  is  not 
necessary  for  the  application  of  AHP  to  source  selection,  we  feel 
it  is  an  area  deserving  of  further  research.  It  is  necessary  to 
investigate  the  theoretical  and  practical  implications  of  such 
translation  methods.  The  impact  of  carrying  a  "red"  evaluation  up 
through  all  levels  of  the  hierarchy  should  also  be  studied. 

An  added  advantage  of  AHP,  and  other  decision  analysis 
methodologies,  in  Step  5  is  the  capability  to  perform  "sensitivity 
analysis."  The  decision  maker  (in  this  case,  the  Source  Selection 
Authority)  is  not  expected  to  accept  the  highest  ranked  alternative 
as  THE  DECISION  after  "running"  the  model  only  once.  Instead,  the 
decision  maker  can  now  investigate  model  characteristics  to 
ascertain  which  have  the  greatest  effect  on  the  decision.  For 
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example,  the  decision  maker  can  perform  sensitivit;^  analysis  to 
answer  such  questions-  as-: 

1.  "Over  what  range  can  the  importance  (weight)'  for  the 
factor  "Manufacturing  Technology,"  vary  while  leaving  the 
ranking  of  contractors  unchanged?" 

2.  "How  much  must  the-  "Systems  Integration"  evaluation 
of  Contractor  3  increase  before  he  becomes  the  preferred 
contractor?" 

This  type  of  analysis  adds  both  insight  into-  the  decision 
problem  and  confidence  in  the  ultimate  decision.  If  for  example, 
the  answer  to  question  2  above  is: 

"Contractor  3,  even  if  he  were  given  the  highest  rating 
in  the  "Systems  Integration,"  would  still  not  be  the 
preferred  contractor." 

the  decision  maker  need  not  be  concerned  about  the  fact  that  he  may 
have  been  unsure  about  whether  Contractor  3  deserved  a  Yellow  or 
Green  rating  on  Systems  Integration. 

Appendix  3  provides  the  complete  hierarchical  model  for  the 
case  study.  We've  assumed  that  the  AHP  was  carefully  and 
thoughtfully  applied  to  arrive  at  the  relative  weights  and  the 
evaluations  shown  here. 


5.4  Advantages  of  the  AHP 

The  proponents  of  the  AHP  have  argued  that  it  has  several  important 
advantages  as  a  decision  analysis  methodology.  These  advantages 
include  the  following. 

Advantage  1;  The  AHP  has  a  simple,  and  intuitively 
appealing  axiomatic  foundation  that  most  decision  makers 
readily  accept.  These  Axioms,  as  simply  stated  by  Harker 
(1989),  are: 

Axiom  1:  Given  any  two  alternatives  (or  criteria), 
the  decision  maker  is  able  to  provide  a  pairwise 
comparison  of  these  alternatives  with  respect  to  any 
criterion  on  a  nine  point  scale. 

Axiom  2:  When  comparing  any  two  alternatives  i,j  the 
decision  maker  never  judges  one  to  be  infinitely 
better  than  the  other  under  any  criterion. 

Axiom  3:  One  can  formulate  the  decision  problem  as  a 
hierarchy . 

Axiom  4:  All  criteria  and  alternatives  which  impact 
the  given  decision  problem  are  represented  in  the 
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hierarchy.  That  is,  all  the  decision  maker's 
intuition  must  be  represented  in  term  of  criteria  and 
alternatives  in  the  structure  and  be  assigned 
priorities  which  are  compatible  with  intuition. 

Advantage  2:  The  AHP  easily  facilitates,  even  fosters, 
subjective  judgement.  Unlike  other  quantitative 
methodologies,  subjectivity  is  explicitly  included  in  the 
AHP. 

Advantage  3;  The  AHP  allows  the  decision  maker  to 
structure  a  complex  decision  problem  and  to  sequentially 
focus  attention  on  small  parts  of  the  problem.  Thus,  the 
AHP  is  in  keeping  with  psychological  research  that 
suggests  the  human  mind  can  only  consider  "7  plus  or 
minus  2”  things  at  a  time  (the  famous  "Miller's  Law," 
Miller,  (1956)). 

Advantage  4:  The  AHP  provides  a  framework  for  thinking 
about,  structuring,  analyzing,  and  synthesizing 
judgements  in  a  complex  decision  problem.  It  generates 
insight  into  the  problem  that  might  not  have  been 
discovered,  identifies  areas  of  the  problem  which  are 
unclear  in  the  decision  maker's  mind,  highlights  where 
additional  information  is  required,  helps  the  decision 
maker  be  consistent  in  his  judgements  and  thinking, 
isolates  and  resolves  areas  of  disagreement  in  the  group 
decision  making  environment.  The  AHP  then  allows  for  a 
rational  and  consistent  synthesis  of  the  "parts"  into  an 
overall  composite  ranking  of  the  alternatives.  As  Saaty 
(1988)  stated: 

The  theory  reflects  what  appears  to  be  an  innate 
method  of  operation  of  the  human  mind.  When 
presented  with  a  multitude  of  elements,  controllable 
or  not,  which  comprise  a  complex  situation,  it 
aggregates  them  into  groups,  according  to  whether 
they  share  certain  properties .  Our  model  of  this 
brain  function  allows  a  repetition  of  this  process, 
in  that  we  consider  these  groups,  or  rather  their 
identifying  common  properties,  as  the  elements  of  a 
new  level  in  the  system... We  were  influenced  by  the 
following  observations: 

(1)  When  we  watch  people  participating  in  the  process 
of  structuring  and  prioritizing  a  hierarchy,  we  find 
that  they  engage  naturally  in  successive  grouping  of 
items  within  levels  and  in  distinguishing  among 
levels  of  complexity, 

(2)  Individuals  informed  about  particular  problems 
may  structure  it  hierarchically  somewhat  differently, 
but  if  their  judgements  are  similar,  their  overall 
answers  tend  to  be  similar.  Also,  the  process  is 
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robust.  In  other  jfords,.  fine  distinctions  within  the 
hierarchy  tend  in.  practice  not  to  be  decisive . 

(3)  In  the  course  of  developing  the  theory  wet^  find  a 
mathematically  reasonable  way  to  handle  judgements, 
(p.  X) 

Advantage  5 :  The  AHP  is  not  intended  to  replace  human 
decision  making,  but  is  instead  to  supplement  it.  Often 
the  results  of  AHP  are  counter-intuitive  which  should 
cause  the  decision  maker  to  reflect  on  his/her  conception 
of  the  problem  formulation  and  his/her  expressed 
pi  <ferences.  This  reflection  should  result  in  further 
insight  and,  ultimately,  a  "better”  decision. 


5.5  Disadvantages  and  controversies  Associated  with  the  AHP 

There  has  recently  been  an  ongoing  debate  regarding  the 
theoretical  soundness,  and  applicability  of  the  AHP.  The  most 
vocal  of  the  opponents  is  Professor  James  S.  Dyer  of  the  University 
of  Texas.  The  proponents'  camp  is  led,  of  course,  by  Professor 
Saaty  with  staunch  allies  Professors  Patrick  T.  Harker  of  The 
Wharton  School,  University  of  Pennsylvania,  and  Luis  G.  Vargas,  a 
colleague  of  Professor  Saaty  at  the  Joseph  M.  Katz  Graduate  School 
of  Business,  University  of  Pennsylvania.  The  debate  is  best 
summarized  by  the  series  of  articles  which  recently  appeared  in 
Management  Science.  (Winkler  (1990),  Dyer  (1990a,  1990b),  Saaty 
(1990),  Harker  and  Vargas  (1990)).  We  shall  attempt  to  briefly 
summarize  those  arguments  in  this  section. 

The  first  criticism  posed  by  Dyer  (1990a)  regards  the 
axiomatic  foundation  of  AHP.  He  said: 

Harker  and  Vargas  claim  that  the  AHP  has  been 
criticized  because  it  lacks  an  axiomatic  foundation,  but 
that  Saaty  (1986)  has  now  provided  the  necessary  axioms 
to  counter  this  deficiency.  The  Axioms  developed  by 
Saaty  (1986)  were  a  significant  contribution  to  the 
theory  of  AHP.  However,  Harker  and  Vargas  miss  an 
important  point  that  can  be  explained  by  a  comparison  of 
Saaty 's  axioms  with  those  of  expected  utility 
theory. . .The  axioms  (of  utility  theory)  have  an  intuitive 
meaning  that  can  be  subjected  to  empirical  tests,  and  the 
preference  function  is  derived  from  these  axioms  rather 
than  assumed  as  a  primitive  notion... In  contrast,  the 
axioms  provided  by  Saaty  (1986)  fail  to  be  motivated  by 
descriptions  of  behavior. 

The  appeal  of  AHP  would  be  strengthened  by  an  effort 
to  link  its  theoretical  basis  to  that  of  classical 
preference  theory,  and  by  providing  a  more  fundamental 
set  of  axioms  descriptive  of  behavior  that  allow  one  to 
derive  the  existence  of  the  ratio  scale...  (pp.  251-252) 
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The  -AHP  supporters  do  not  agree  with  this  criticism.  The 
arguments  put  forth  in  Barker  and  Vargas  (1990)  basically  assert 
that  the  axiom  system  for  AHP  is  no  less  logical,  no  less 
descriptive  of  actual  behavior  than  the  axioms  of  utility  theory. 

Dyer  and  others  have  shown  that  "rank  reversal”  can  occur  in 
the  AHP  when  an  additional  alternative,  similar  to  others  already 
included,  is  added  to  an  AHP  model.  ’’Rank  reversal”  occurs  when, 
for  example,  the  alternatives  are  ranked  A,  B,  then  C  initially, 
but  are  ranked  A,  D,  C,  and  B  after  alternative  D  has  been  added. 
Notice  that  the  relative  rankings  of  aJ.ternatives  B  and  C  have 
reversed.  Because  of  this.  Dyer  concludes  that  the  ranking  of 
alternatives  as  determined  by  the  AHP  are,  in  essence,  arbitrary. 
He  states  (Dyer  (1990a)); 

The  difficulty  can  be  simply  stated  as  follows:  The 
ranking  of  alternatives  determined  by  the  AHP  may  be 
altered  by  the  addition  of  another  alternative  for 
consideration.  This  characteristic  of  the  methodology 
has  been  well  known  for  years  and  has  been  discussed  in  a 
number  of  articles  by  critics  and  by  proponents  of  the 
AHP... The  real  issue,  however^  is  not  the  phenomenon  of 
rank  reversal  per  se.  Rather,  rank  reversal  is  a  symptom 
of  a  much  more  profound  problem  with  AHP:  the  rankings 
provided  by  the  methodology  are  arbitrary,  (p.  252) 

The  AHP  proponents  counter  this  argument  by  showing  that  when 
using  "absolute  measurement,”  instead  of  "relative  measurement," 
rank  reversal  cannot  occur.  "Absolute  measurement”  does  not 
require  the  pairwise  comparisons  of  alternatives  with  respect  of 
each  criterion.  Instead,  each  alternative  is  rated  against  each 
criterion  after  the  relative  weights  of  the  ratings  have  been 
determined.  Furthermore,  the  AHP  camp  argues  that  the  examples 
exhibiting  rank  reversal  have  added  an  alternative  which  is 
essentially  a  "copy"  of  an  alternative  included  in  the  original 
model.  They  contend  this  creates  an  artificial  example  that  does 
not  occur  in  realistic  decision  problems.  Finally,  Saaty  (1990) 
contends  the  reversal  is  a  legitimate  phenomenon,  truly  reflective 
of  the  decision  process  at  hand.  He  states: 

In  a  relative  measurement  the  preference  for  an 
alternative  is  determined  by  all  other  alternatives. 

In  this  sense  the  alternatives  are  not  independent  from 
each  other  for  the  determination  of  their  priorities. 

This  dependence  among  the  alternatives  may  be 
reinterpreted  as  a  rescaling  of  the  priorities  of  the 
criteria  depending  on  how  many  alternatives  there  are. 

Stated  differently,  the  presence  or  absence  of  an 
alternative  in  relative  measurement  introduces 
additional  information  regarding  the  dominance  of  that 
alternative  with  respect  to  the  other  alternatives, 
whatever  their  number  is.  This  information  arises  from 
change  in  the  structure  of  the  decision  problem  like 
adding  or  deleting  a  variable  in  a  linear  programming 
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problem.  The  new  optimum  must  be  calculated  from  the 
start  and  would  not  usually  coincide  with  the  previous 
optimum  on  some  of  the  variables.  The  new  information 
in  the  decision  problem  is  represented  by  a  s^actural 
criterion,  which  instead  of  being  compared  wit^  the  old 
criteria  from  which  it  is  fundamentally  different, 
operates  as  a  transformation  on  the  priorities' {of  those 
criteria  by  rescaling  them  according  to  fhe  new 
information.  This  is  not  like  anything  encounpered  in 
utility  theory.  It  is  new  ,and  logical,  but  certainly 
not  arbitrary.  By  changing  the  priorities  of  the 
criteria,  the  collection  of  old  and  new  alternatives  or 
simply  the  old  collection  with  a  missing  alternative 
may  have  a  new  ranking  not  compatible  with  the  cld  one. 

(p.  264) 

Dyer  concludes  with  the  suggestion 

...the  actual  solution  to  this  problem  is  relatively 
simple,  and  is  based  on  the  synthesis  of  the  AHP 
assessment  methodologies  with  the  theory  of 
multiattribute  utility  theory  (MAUT) . 

At  a  higher  level  of  abstraction  we  assume  that 
researchers  in  '•utility  theory"  and  in  AHP 
methodologies  are  attempting  to  model  the  preferences 
of  a  decision  maker  so  that  the  rankings  of 
alternatives  produced  by  these  approaches  reflect  these 
preferences.  Therefore,  it  seems  reasonable  to 
conclude  that  both  fields  would  benefit  from  efforts  to 
synthesize  these  two  approaches  to  the  same  problem... 

We  conclude  that  much  more  is  to  be  gained  from  a 
synthesis  of  AHP  and  MAUT  than  from  efforts  to  maintain 
them  as  separate  areas  of  research  and  application. 

(pp.  251-236) 

Despite  the  fact  that  the  debate  rages  on,  theoretical 
progress  and  successful  application  of  both  methodologies 
continues.  From  a  pragmatic  point  of  view,  the  true  test  of  the 
viability  of  any  decision  analysis  methodology  is  whether  or  not  it 
assists  the  decision  maker  in  structuring,  understanding  and 
resolving  complex  decision  problems. 
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6.  Conclusions  and  Reconunendations  for  Further  Research 

In  this  paper,  we  have  investigated  the  application  of  two 
established  decision  analysis  methodologies.  Multiattribute  Utility 
Theory  and  the  Analytic  Hierarchy  Process,  to  the  problem  of  source 
selection,  an  important  and  recurring  problem  in  the  Department  of 
Defense.  The  scope  of  our  discussion  was  limited  to  Air  Force 
systems  acquisition  projects.  Source  selection,  MAUT,  and  the  AHP 
were  briefly  explained.  The  decision  methodologies  were  then 
applied  to  an  actual  source  selection  problem,  although  the 
evaluations  used  were  fictitious. 

It  was  shown  that  both  decision  analysis  methodologies  can  be 
applied  to  this  important  class  of  problems.  We  conclude: 

1.  Both  the  AHP  and  MAUT  provide  a  rational,  consistent, 
communicable  and  logically  defensible  means  of 
synthesizing  the  many  individual  judgements  and 
evaluations  into  a  prioritization  of  contractors. 
Moreover,  this  synthesis  is  accomplished  very  efficiently 
using  readily  available  microcomputers  and  software. 

2.  The  area,  item,  factor,  and  subfactor  structure 
provided  in  the  Source  Selection  Plan  is  easily  and 
directly  translated  into  a  hierarchical  model. 

3.  Sensitivity  analysis  results  in  significantly 
increased  understanding  of  the  problem  and  its 
characteristics,  insight  not  currently  attainable  using 
the  current  procedures. 

5.  The  application  of  MAUT  requires  extensive  assistance 
of  and  interaction  with  a  trained,  experienced  decision 
analyst,  whereas  the  AHP  can  be  easily  used  with  little 
training . 

We  are  not  the  first  to  come  to  these  or  similar  conclusions. 
It  is  our  opinion  that  the  application  of  decision  analysis  to 
source  selection  holds  great  promise. 

Further  research  might  be  directed  at  the  following 
areas: 

1.  Development  of  a  user  friendly  software  package  which 
implements  decision  analysis  specifically  for  the  source 
selection  problem  domain. 

2.  Investigation  of  the  applicability  of  decision 
analysis  to  source  selection  in  the  other  military 
services. 

3.  A  theoretically  sound  means  of  translating  composite 
priorities  into  the  familiar  color  scheme  currently 
employed. 
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4.  Development  of  a  program  to  educate  acquisition 
personnel  about  the  benefits  and.  use  of  decision*^  analysis 
in  source  selection. 

5.  Development  of  a  test  program  to  further  assess,  the 
efficacy  of  applying  decision,  analysis  to,  source 
selection. 

6.  A  more  extensive  survey  of  previous  applications  of 
the  decision  analysis  to  systems,  acquisitiom  and  the 
benefits  obtained. 

7.  Methods  of  combining  several  individual  analyses  into 
a  single  decision. 

8.  Psychological  and  behavioral  aspects  of  both 
individual  and  group  decision  making  and  the  implications 
for  employing  the  decision  analysis  in  source  selection. 

9.  Investigation  of  the  decision  analysis  methodology 
employed  at  Stanford  University  under  the  direction  of  R. 
A.  Howard  (c.f.  Howard  and  Matheson  (1984),  and  Tatman 
(1989)) . 


This  research  was  funded  by  the  Defense  Systems  Management 
College,  Washington  D.C.,  and  the  Systems  Acquisition  Research 
Group,  United  States  Air  Force  Academy. 
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APPENDIX  1:  Source  Selection  Case  Study 


This  Appendix  contains  information  obtained  from  an  actual  Air 
Force  Systems  Command  source  selection.  Pages  1-2  through  1-7  are 
pages  from  Source  Selection  Plan  which  provide  the  areas,  items, 
factors  and  assessment  criteria  for  this  source  selection.  Note 
that  information  which  might  reveal  the  identity  of  the  weapon 
system  under  consideration  has  been  blacked  out.  These  pages 
provide  the  information  from  which  the  MAUT  and  the  AHP  models  were 
constructed. 

The  remaining  pages  of  Appendix  1  (pp.  1-8  through  1-28)  are  the 
fictitious  evaluations  of  four  contractors  proposals.  Actual 
evaluations  were  not  provided  to  us  for  this  case,  however  this 
was  not  critical  to  this  research.  Actual  evaluations  could  easily 
be  substituted  for  the  fictitious  ones  used  herein  with  no 
significant  impact  on  the  application  of  MAUT  or  AHP  to  source 
selection,  the  resultant  models,  or  the  conclusions  and 
recommendations  reached  in  this  study.  Also  note  that  not  all 
evaluation  matrices  are  provided.  We  provide  only  enough  of  these 
to  give  the  reader  an  understanding  of  how  the  decision  analysis 
models  are  constructed. 
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each  CR,  DR,  and  including 
each. 


d  Mif  in 


r  rSpohses  and  final  disposition  of 


6.  Lessons  Learned:  At  the  conclusion  of  source  selection  activi¬ 
ties,  the  adalnlstratlve  group  td.ll  develop  and  publish,  any  lessons  learned, 
subject  to  approval  by  the  chairman  of  the  SSEB;  in  a- single  bound  document. 

V.  EVALOATIOH  CRITERIA'  '■  • 

A.  ■  Introduction: 

^^^^^^^^v^h^Gove^enc-provlded||||||{statement  of  tfork  (S0U),imm 

other  in  Section  L  of  the  RFP,  the 

T)tterora  wiii  propos^^^^rogram  they  believe  best  meets  system  requirements,' 
demonstrates  that  th^HP concept  is  technically  feasible  and  reduces  program 
risks  to  the  point  acceptable  to  proceed  into  the  Full  Scale  Development  (FSD) 
phase  of  the  program*.  The  offerors  will  also  propose  the  price  of  the  Demon¬ 
stration/Validation  (Dem/Val)  effort  and  their  preliminary  cost  estimates  for 

•  FSD,  production,'  and  Operations  and  Support  (O&S),  along  with  methodology  for 

■  calculating  these  estimates.  .  .. 

'•  *•  Basis  for  Award;  'I  i  ’• 

The  planned  selection  of  four  contractors  for  the  Dem/Val  phase  of 
'^11  be  made  on  the  basis  of  an  Integrated  assessment  of  the 
proposals  submitted  in  response  to  the  RFP  and  the  proposed  terms  and  con¬ 
ditions  contained  within  the  executed  (negotiated)  contract.  In  making  this 
integrated  assessment,  the  following  specific  ereas.will  be  evaluated  (listed- 
in  descending  order  of  importance): 

■  r':%: -m' ^ ....  • 

*•  .  Technical; > ‘  f  >■ 

■  ■  Supportablllty';’. '  .  _ 

m  '  '  '  '  :V- '  .  . 

^  c.  Management  Capability;  and,  '• 

^d.  Cost. 


.  ^d.  Cost.  - 

The  following  assessment  criteria  (ranked  equally)  will  be  used  to  rate  each 
area  of  the  proposal  evaluation:  '  '  . 


a.  Soundness  of  Approach;  •  • 

b.  Understanding  of  the  Problem; 

c.  Compliance  with  Requirements;  and 

d.  Past  Performance, 


The  Government  will  review  each  proposal  against  the  above  criteria  using  both 
internal  and  offeror-provided  data  in  the  assessment*  The  evaluation  will 
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Include  an  assessment  of  the  risks  involved  with  the  design  approach  proposed 
and  the  means  proposed  to  reduce  these  risks.  Throughout  the  evaluation,  the  . 
Government  will  consider  “correction  poteatlal“  when  a  deficiency  is  identi¬ 
fied  in  the  proposal.  Proposals  unrealistic  in  terms  of  technical,  support-  ' 
ability,  or  schedule  commitments,  or  unrealistically  low  in  cost  or  price, 
will  be  deemed  reflective  of  an  Inherent  lack-  of  technical  competence  or  indi¬ 
cative  of  a  failure  to  comprehend  the  complexity  and  risks  of  the  contract 
requirements.  These  conditions  may  be  grounds  for  rejection  of  the  proposal. 
The  Source  Selection  Authority  will  make  a  determination  of 'the  overall  value 
of  each  proposed  system  approach  judged  in  terms  of  its  potential  to  best 
satisfy  the  needs  of  the  Government,  all  factors  considered..  Subjective 
judgment  on  the  part  of  the.  Government  is  Implicit  in  the  evaluation  process. 
The  Government  reserves  the  right  to  award  contracts' at  other  than  the  lowest 
proposed  Dem/Val  price,  lowest  estimated  FSD  price,  or  lowest  life  cycle  cost. 


1.  .A  detailed  evaluation  will  be  made,  of  each  offeror's  oMoosal  to 
provide  information  to  the  SSA  for  use  in  source  selection.  ■  The^Hpoem/Val 
proposal  evaluation  will  Include  consideration  of  the  areas  descrioea  below,., 
in  order  of  relative  importance.'  Thej^flmust  meet  the  mission  performance 
requirements,-  Rellabllity/Halntalnabillty  (R/H)  requirements,  and  must  be  a 
producible  design.  To  clearly  set  the  Importance  of  these  required  features, 
R/M,  Produclblllty/Quality  Engineering,  and  System  Engineering  are  the  highest 
ranked  items  -  under  the-  technical  area.  -  The  remaining  items  are  equally 
ranked.  Excluding  the  Cost  area,  items  within  the  other  areas  (Supportabllity ' 
and  Management  Capability)  are  equally  Important  within  their  respective 
areas.  All  factors  (or  items  which  do  not  have  factors)  will  be  measured 
against  specific  standards*.  Each  area  is  rated  using  all. four  assessment  cri¬ 
teria.  ■  “  •  ■  ... -l' 

'I  a.  '  TECHNICAL  AREA,  .  <  -V 

■  ■  ■  ; 

The  quality  of  the  contractor  proposed  procedures  for  accom¬ 
plishing  design  synthesis  and  validation  during  the  Dem/Val  program  phase  Is 
of  primary  importance.  The  evaluation  will  focus  on  a  combination  of  the 
following  elements:  (1)  weapon  system  concept  and  its  threat-driven  charac¬ 
teristics,  (2)  the  engineering  plans  for  refining  the  weapon  system  designs  to 
provide  superior  combat  capability  for  world-wide  deployment  based  upon 
evolving  operational,  reliability,  maintainability,  and  producibllity  require¬ 
ments,  and  (3)  the  degree  to  which  the  proposed  engineering  planning  supports 
the  technologies  used  in  establishing  a  system  configuration.  The  proposal 
must  demonstrate  and  convey  clearly  an  understanding  of  the  technical  achieve¬ 
ments  required  by  the  SRD  and  SOV  and  the  level  .of  effort  needed  to  achieve 
the  required  capability.  Engineering  program  planning  will  be  assessed  for 
reasonableness  and  use  of  acceptable  design  practices.  The  following  items 
and  factors  will  be  used  in  proposal  evaluation  In  the  technical  area: 
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”  Rellablllty/Malntainabillty  Item 

—  Reliability  ■  ■ 

“Maintainability  ■'  y'  •*' 

-  Producibility/Ouality  Engineering  Item 

■  ■  —  Produclbllity  in  Design  •  ■  ^  ' 

“  Quality  in  Design-  '  -l"  ■ «. 

Manufacturing  Technology  ' 

:  —  Manufacturing  Research. 

--i' ' 


"  System  Engineering  Item  ....  . 

System  Effectlvenesn •'•■"'.'f  !. 
.Ornament  Integratlony-.;'  7;V ^  ; 

.  *■  .  Systems  Integration  !Vr’^ 

—•  Computer  Resources.-.' .'j,  • 

’  ,  '  ■ . ■/.  v.-v  ■V'’V‘.-i‘'’Vr  f ' 

•  '  •  '  '  '  ■  -  Flight  Systems  Item"-;"  ’.t.'"’--  '•  . 


Flight  Systems  Item 

'  --  Structures^ 
Propulsion 
Stability  and 


*  »  •  •  •  •  V  .  ^  \  .  ••  •  •••  . 

.it .. . .  V' ..1.  ...  j*:  »2.  .  .• 

%*r  •  •  :  •  •  .  •  *  V  i'  .  *  '  ”  '  • 

'•  ;/ -V,?..'  v.'.**  '  ■  ‘  '■*•'•••  ■  ■••• 

..  .  - /  and  Cbntro^.'^''•>^:^^'•■^CV.•V■./.^‘;.y  , 

Aerodynamics  and  Performanct  '-'V'  ' 

'.‘.V- ““Flight  Equipment  and  Subsystems':':- ‘ 

.'  .  .  ■  --j  .  .  .  ; .  ''.v  ■ 

.  ».  -  ••  Avlon-fr*  ...-V  -'i-'.  .  .. 


Avionics  Item 

1  ♦.  *t  *  ’* 

•  -“  Comnunlcatlons/Navlgatlon. ■ ' 

•—  Offensive  Functions.  —  '  ' 

--  Defensive  Functions  '  •  '• 

•  Architecture 
-“  Controls  and  Dl'splays','  " 

—  Observables 

“  Support  Systems  Item  ' 

~  Life  Support/Escape  Systems 
—  Crew  Systems 
■—  Maintenance  Systems. 

--  Human  Factors 
Training  Analysis 


b.  SUPPORTABILITY  AREA. ' 


^portability  will  be  evaluated  la  terms  of  the  offeror’s 
approach  to  the  management,  technical,  and  operations  concepts.  The 
approach  for  assessing  the  Impact  on  supportability  of  the  design  concept  will 
be  evaluated  by  the  technical  panel.  The  management  and  operations  aspects  of 
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supportablllty  will  be  evaluated  as  defined  below.  The  offeror  must  present 
evidence  of  the  technical  capability,  managerial  capability  and  corporate  com-' 
mltment  needed  to  vigorously  plan.  Implement,  monitor  and  control  the  support- 
ability  program  required  to  design,  demonstrate  and  produce  an  affordable, 
supportable  and  ef f active |||||||  weapon  system.  The  offeror's  proposed  approach 
and  thoroughness  in  conduct?^  LCC  tradeoff  analyses  for  Dem/Val  will  also  be 
evaluated  in  this  area.  The  following  items  and  factors  will  be  used  in  pro¬ 
posal  evaluation  in  the  supportabillty  area:  •  .  ,  . 

-  Organizational  Structure  Item  ‘ 

•  •  ~  Management  Policies,  Practices,  and  Planning  Item  ' 

"  .  *  *  .  <  ♦  ,  ■  'I  * 

=  ■  —  ILS  Program 

'  ;  —  LCC  Program  ..  .  ' 

.  “  LSA  Program  '  _  1  •  i  ■ 

—  Lessons  Learned  Program  .  - 

-  Supportabillty  Program  Implementation  Item 

—  Influence  on  Design  Process  '■ 

,  —  Integration  of  SupportabiWty  Efforts 

-  Logistics  Capabilities  Item  '.V-- 

—  Definition  of  Logistics  Issues- 
.■  7-  Tools  and  Techniques  for  Supportabllity  Evaluation .  ’ ' 

J  C.  MANAGEMENT  CAPABILITY  AREA,-. 

The  contractor's  organizational  structure,  management  plans, 
and  overall  program  schedule  will  be  evaluated.  Consideration  will  be  given 
to  the  distribution  of  responsibility  within  the  organization,  supervisory 
authority  over  personnel,  the  location  and  responsibility  of  key  personnel 
in  the  organizational  structure,  internal  review-  procedures  of  management ' 
activities  and  decisions,  participation  of  functional  organizations  in  the  ' 
design  process,  interface  with  and  management  visibility  provided  to  the  Gov¬ 
ernment  of  internal  and  external  organizations,  and  individual  plan  schedules 
and  their  relationship  with  total  program  schedule.  The  management  capability 
evaluation  will  Include  assessment  of  the  following  items  and  factors: 

-  Program  Management  Item 

^  —  Configuration  Management  Planning  ‘  . 

—  Data  Management 
—  Organization 

—  Management  Control  and  Information 
—  Contingency  Planning 
—  Contractor  Interfaces 


Future  Competition  Itea: 


■  •  ~  Syste'mg.  Test.  Item. 

--  Hanuf acturtng:  Item'  • 

—-Manufacturing^  Plannln^^,. 

—  Manufacturing  Capability .  '  . 

—  Manufacturing  Cost  Reduction' 

-  Quality  Assurance  Item 

•,  —  Quality' System  '  ■  • 

—  Initial,  Quality  Planning 

-  System  Safety  Item  ‘ 

■  -  Training  Systems.  Mana<i>ement  Item 

-  Security  Item- 

.  .  —  Personnel/Physical  Security  and  Classification  )^t 

■  —  TEMPEST/COMSEC/OPSEC: 

■  '  ■  :  —  System,  Security  Engineering  Management  Program 

'^d.  CQSTAREA. 

'  In  the ^P|;0em/Val  phase  source  8eleetion»  the  cos^panel**' 
objective*  are  to  evaluate  the  affordability  of  each  offeror’s  ^HH system 
approach/cbncept  for  FSD,  production,  and.  Q&S  as  weH  as.  the  methodoT^y  used 
in  constructing  their  estimates*  The  proposed  Dem/Val  contract  price, 
although  it  will  be  evaluated  for  reasonableness  and  consistency  by  the  conr 
tracting  negotiators  and  auditors,  is  of  secondary  Importance,  Each  offeror 
shall  provide  their  best  cost  estimate  for  FSD,  product^n,  and  O&S  using  the 
format,  definitions  and  ground  rules  prescribed'  in  the  ^^^Dem/Val  Cost  Annex* 
The  cost  evaluation  will  be  based  on  an  assessment  of  ea»  offeror's  proposed 
cost,  and  the  government's  estimate  of  most  probable .total  life  cycle  cost  for 
the  planned  force  structure  for  each  offeror  considering: 

-  Realism;  Costs  and,  scope  of  work  are  compatible. 

-  Reasonableness;  Acceptable  estimating  methodology* 

*  -  Completeness;  Responsiveness  .  in  providing  all  RFP 

requirements,  SOW  items,  and  traceability  of  estimates* 

2*  The  overall  evaluation  of  each  proposal  may  Include  on-slte 
Inspections  and  results  of  pre-award  surveys  to  provide  Information  to  the 
Source  Selection  Authority  regarding  offerors  current  and  future  capability  to 
perform  all  aspects  of  the  program*  Risk  assessment  associated  with  the  major 


areas  of  the  prograa  will  be  accomplished.  In  assessing  risk,  an  Independent 
judgment  of  the  probability  of  success,  the  impact  of  failure,  and  the  alter-* 
natives  available  to  meet  the  requirements  will  be  considered. 


VI.  ACQUISITION  STRATEGY 


The  overall  acquisition  strategy  follows  recommendations  approved  by  the 
AFSC  Business  Strategy  Panel  (BSP)  on  6  Aug  84  as  refined  by  the  Air  Force 
System  Acquisition  Review  Council  ■  (AFSARC)  on  13  Nov  84  and  the  AFSC/ASD 
Solicitation  Review  Panel  (SRP)  on  15  Jan  85.  This  plan  updates  and  amplifies 


the  strategy  discussed  in  thei 
lighted  prograa  strategy  Include! 


High- 


A.  Fixed  Price  contracts  are  planned  for  Dem/Val  to  limit  the  govern¬ 
ment’s  cost  risk  in  the  highly  compet^^e  environment  which  precedes  a  major 
downselectlon,  ,(l*e. ,  selection  of  anfljB  FSD  contractor).  - 


U.  Solicitations,  id.ll  include  anticipated  funding  profiles  by  fiscal 
years.  .  .  . .  ,  .  •  i  ,  ^ 


E.  Cost  reporting  sufficient  to  evaluate ■’Den/Val  contractor’s  expen- 
ditures  under  the  FFP  contract  will  be  required..  . 


F.  Requirement  for  explicit  planning  to  assure  adequate  competition  for 
critical  subsystems  will  be  available  by  the  time  the||||Bsystem  enters  the 
production  phase  (e.g. ,  dual  sources,  unlimited  right^xn  technical  datai^ 
etc.). 

VII.  SCHEDULE  OF  EVENTS 

ASD  Business  Strategy  Panel  Convened 
ASD  Acquisition  Management  Panel  Convened 
AFSC  Business  Strategy  Panel  Convened 
Draft  RFP  Released 

.  SSA  Approves  the  Source  Selection  Plan 
RFP  Released 

SSAC  Formally  Established 
Standards  Prepared/Approved 
•  Proposals  Received  -  Evaluation  Starts 
Initial  Evaluation  Completed 
Competitive  Range  Determination 
SSSB  Initial  Evaluation  and  Competitive 
Range  Briefing  to  SSAC 
Evaluation  Completed 
Discussions  Completed 
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Overall  Assessment 


Supportablllty 
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Management  Capability 
_ Contractor  A 


Supportablllty 


Management  Capability 
_ Contractor  B 


as 


Supportablllty 


Management  Capability 


••  Cost 
Contractor 


Management  Capability 


Appendix  2  (MAUT  model) 


Appendix  2  contains  printouts  from  EXPRESSION  TREE,  utility 
values,  probability  assessments,  and  a  generic  decision  tree. 


I.  EXPRESSION  TREE  Printouts 

The  TREE  EVALUATION  printout  is  the  evaluation  of  the  model  using 
expected  utility  theory.  The  recommended  alternative  is 
Contractor  C.  Sensitivity  analyses  (see  below)  indicate 
Contractor  C  is  still  preferred. 

NODE;  1  (Decision)  C.E.;  0.858  (Risk  Neutral — Inicreasing) 
NODE  NAME;  SS  DECISION 


BRANCH  LABEL 

BRANCH  VALUE 

CERTAINTY  EQUIVALENT 

CONTRACTOR  A 

0.000 

0.810 

CONTRACTOR  B 

0.000 

0.722 

CONTRACTOR  C 

0.000 

0.858  <- 

CONTRACTOR  D 

0.000 

0.775 

SENSITIVITY  ANALYSIS  is  a  "what  if  analysis. 

Vary  Probability  Variable;  P_TECHB_A 

PROBABILITY 

CERTAINTY  EQUIVALENT 

PREFERRED  DECISION 

VALUE 

[Node  1] 

0.00000 

0.858 

CONTRACTOR  C 

0.06000 

0.858 

CONTRACTOR  C 

0.12000 

0.858 

CONTRACTOR  C 

0.18000 

0.858 

CONTRACTOR  C 

0.24000 

0.858 

CONTRACTOR  C 

0.30000 

0.858 

CONTRACTOR  C 

0.36000 

0.858 

CONTRACTOR  C 

0.42000 

0.858 

CONTRACTOR  C 

0.48000 

0.858 

CONTRACTOR  C 

0.54000 

0.858 

CONTRACTOR  C 

Vary  Probability  Variable:  P_TECHG_A 

PROBABILITY 

CERTAINTY  EQUIVALENT 

PREFERRED  DECISION 

VALUE 

[Node  1] 

0.00000 

0.858 

CONTRACTOR  C 

0.07400 

0.858 

CONTRACTOR  C 

0.14800 

0.858 

CONTRACTOR  C 

0.22200 

0.858 

CONTRACTOR  C 

0.29600 

0.858 

CONTRACTOR  C 

0.37000 

0.858 

CONTRACTOR  C 

0.44400 

0.858 

CONTRACTOR  C 

0.51800 

0.858 

CONTRACTOR  C 

0.59200 

0.858 

CONTRACTOR  C 

0.66600 

0.858 

CONTRACTOR  C 

0.74000 

0.858 

CONTRACTOR  C 
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Vary  Probadiility  Variable:  P_.TECH(^B 


PROBABILITY 

CERTAINTY  EQUIVALENT 

RREFERI^D  DECISION 

VALUE 

[Node  .1] 

• 

0.00000 

0.858 

CONTRACTOR  C 

.0.09900 

0.858 

CONTI^CTOR  C 

.0.19800 

0..858 

CONTiaCTOR  C 

0.29700 

0.858 

CONTidkCTOR  C 

0.39600 

0.858 

CONTlilCTOR  C 

0.49500 

0.858 

CONTRACTOR  C 

0.59400 

0.858 

CONTRACTOR  C 

0.69300 

0.858 

CONTI^CTOR  C 

0.7,9200 

0.858 

CONTRACTOR  C 

0.89100 

0.858 

CONTWICTOR  C 

0.99000 

0.858 

CONTRACTOR  C 

Vary  Probability  Variable:  P_TECHB_C 


PROBABILITY 

CERTAINTY  EQUIVALENT 

PREFERRED  DECISION 

VALUE 

[Node  1] 

0.00000 

0.849 

CONTRACTOR  C 

0.02900 

0.850 

CONTRACTOR  C 

0.05800 

0.852 

CONTRACTOR  C 

0.08700 

0.854 

CONTRACTOR  C 

0.11600 

0.856 

CONTRACTOR  C 

0.14500 

0.857 

CONTRACTOR  C 

0,17400 

0.859 

CONTRACTOR  C 

0.20300 

0.861 

CONTRACTOR  C 

0.23200 

0.863 

CONTRACTOR  C 

0.26100 

0.864 

CONTRACTOR  C 

0.29000 

0.866 

CONTRACTOR  C 

Vary  Probability  Variable:  P_TECHG__C 


PROBABILITY 

CERTAINTY  EQUIVALENT 

PREFERRED  DECISION 

VALUE 

[Node  Ij 

0.00000 

0.830 

CONTRACTOR  C 

0.08400 

0.833 

CONTRACTOR  C 

0.16800 

0.836 

CONTRACTOR  C 

0.25200 

0.840 

CONTRACTOR  C 

0.33600 

0.843 

CONTRACTOR  C 

0.42000 

0..  847 

CONTRACTOR  C 

0.50400 

0,850 

CONTRACTOR  C 

0.58800 

0.853 

CONTRACTOR  C 

0.67200 

0.857 

CONTRACTOR  C 

0.75600 

0.860 

CONTRACTOR  C 

0.84000 

0.863 

CONTRACTOR  C 
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Vary  Probability  Variable:  P_TECHB_D 


PROBABILITY 

CERTAINTY  EQUIVALENT 

PREFERRED  DECISION 

VALUE 

[Node  1] 

0.00000 

0.858 

CONTRACTOR  C 

0.09900 

0.858 

CONTRACTOR  C 

0.19800 

0.858 

CONTRACTOR  C 

0.29700 

0.858 

CONTRACTOR  C 

0.39600 

0.858 

CONTRACTOR  C 

0.49500 

0.858 

CONTRACTOR  C 

0.59400 

0.858 

CONTRACTOR  C 

0.69300 

0.858 

CONTRACTOR  C 

0.79200 

0.858 

CONTRACTOR  C 

0.89100 

0.858 

CONTRACTOR  C 

0.99000 

0.858 

CONTRACTOR  C 

The  TREE  SUMMARY  printout  is  a  description  of  each  node. 

1-D:  SS_DECISION 

CONTRACTOR  A  CONTRACTOR  B  CONTRACTOR  C 

CONTRACTOR  D 

0.000  0.000  0.000  0.000 

2  11  21  31 


Note;  1-D  is  a  decision  node  and  is  arbitrarily  labeled  as  node 
1.  There  are  four  contractors  (A,  B,  C,  and  D)  in  this  source 
selection  decision.  The  0.000  indicates  that  there  is  no  branch 
value  for  the  decision.  The  numbers,  2,  11,  21,  and  31  indicate 
the  next  node. 


2-C;  TECHNICAL 
BLUE 
1.000 
3 

0.25000 


GREEN 

0.950 

3 

0.40000 


YELLOW 

0.850 

3 

0.35000 


Note:  2-C  is  a  chance  node  and  is  labeled  as  node  2.  The 
uncertain  variable  is  for  the  technical  area  and  has  three 
possible  outcomes,  namely,  blue,  green,  and  yellow.  The  numbers 
1.000,  0.950,  and  0.850  indicate  the  utility  values  for  technical 
for  outcomes  of  blue,  green,  and  yellow,  respectively.  The 
number  3  is  the  next  node.  The  likelihoods  of  a  blue,  green,  and 
yellow  are  0.25,  0.40,  and  0.35.  The  following  are  other  chance 
nodes  for  contractors  A,  B,  C,  and  D. 

3-C;  SUPPORTABILITY 


BLUE 

GREEN 

YELLOW 

1.000 

0.900 

0.500 

4 

4 

4 

0.15000 

0.55000 

0.30000 

MANAGEMENT 

BLUE 

GREEN 

YELLOW 

1.000 

0.800 

0.400 

5 

5 

5 

0.40000 

0.55000 

0.05000 
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3-iC: 


12-^: 


13-C; 


14-C: 


21-C: 


22-C: 


23-C; 


24-C: 


31-C; 


COST 

GREEN 

YELLOW 

0.650 

0.:350 

41 

41 

0.35000 

0.65000 

TEdHNICAL 

GREEN 

YELLOW 

0.950 

0.850 

12 

;i2 

0.50000 

:0.;50000 

SUPPORTABILITY 

GREEN 

YELLOW 

0.900 

:0.500 

13 

J.-3 

0.'30000 

0.70000 

MANAGEMENT 

GREEN 

YELLOW 

0.800 

0.400 

14 

14 

0.65000 

.0.-35000 

COST 

GREEN 

YELLOW 

0.650 

.0.350 

41 

41 

0.50000 

0.50000 

TECHNICAL 

BLUE 

GREEN 

YELLOW 

1.000 

0.950 

0.'850 

22 

'22 

22 

0.15000 

0.70000 

0.15000 

SUPPORTABtLITY 

BLUE 

GREEN 

1.000 

0.900 

23 

23 

0.70000 

0.30000 

MANAGEMENT 

BLUE 

GREEN 

YELLOW 

1.000 

0.800 

0.400 

24 

24 

24 

0.45000 

0  .<45000 

0.10000 

COST 

GREEN 

YELLOW 

0.650 

0.350 

41 

41 

0.35000 

0. '65000 

TECHNICAL 

BLUE 

GREEN 

1.000 

0.950 

32 

32 

0.5000C 

0.50000 

.2-4 


32-C:  SUPPORTABILITY 
GREEN 
0.900 
33 

0.70000 


YELLOW 

0.500 

33 

0.30000 


33-C:  MANAGEMENT 
GREEN 
0.800 
34 

0.55000 


YELLOW 

0.400 

34 

0.45000 


34-C:  COST 

GREEN  YELLOW 

0.650  0.350 

41  41 

0.30000  0.70000 

41-E:  KT*TECHNICAL  +  KS*SUPPORTABILITY  +  KM*MANAGEMENT  +  KC*COST 


Note:  41-E  is  the  end  node  and  is  labeled  as  41.  This  end  point 
represents  the  following  additive  utility  function: 


U(t,  s,  m,  c)  =  Kt*U(t)  +  Ks*U(s)  +  Kin*U(m)  +  Kc*U(c) 


The  SHOW  NODE  printout  is  another  way  to  express  nodes  in 
EXPRESSION  TREE,  see  below. 


Node  Number  :  1 

Node  Name  :  SS_DECISION 

Branch  Label  Branch 

Value 

Node  Type  :  Decision 
Number  Of  Branches  :  4 

Nxt 

CONTRACTOR  A 

0.000 

2 

CONTRACTOR  B 

0.000 

11 

CONTRACTOR  C 

0.000 

21 

CONTRACTOR  D 

0.000 

31 

Node  Number  :  2 
Node  Name  :  TECHNICAL 


Node  Type  :  Chance 
Number  Of  Branches  :  3 


Branch  Label  Branch  Value  Nxt 


Branch  Probability  Expression 


BLUE 

GREEN 

YELLOW 


1.000 

0.950 

0.850 


3  P_TECHB_A 

3  P_TECHG_A 

3  1-P  TECHB  A-P  TECHG  A 


Node  Number  :  3 

Node  Name  :  SUPPORTABILITY 


Node  Type  ;  Chance 
Number  Of  Branches  :  3 


Branch  Label 


Branch  Value  Nxt 


Branch  Probability  Expression 


BLUE 

GREEN 

YELLOW 


1.000  4  .15 

0.900  4  .55 

0.500  4  .3 
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Node  Number  :  4 

Node  Name  MANAGEMENT 

Branch  Label  Branch  Value 

Nxt 

Node^  Type.  :  Chance 

Number  Of  Branches  :  3 

Branch  Probability  Expressiop 

BLUE 

l.OOQ 

5 

.4 

GREEN 

0.800. 

5 

.55 

YELLOW 

0.400 

5 

.05: 

Node  Number  :  5 

Node  Type  :  (%ance 

Node  Name  COST 

Number  Of  Branches  :  2 

Branch  Label  Branch  Value 

Nxt 

Branch  Probability  Expression 

GREEN 

0.650 

41 

.35 

YELLOW 

0.350 

41 

.65 

Node  Number  :  41 

Node  Type  :  End 

END  NODE  EXPRESSION 

KT*TECHNICAL  +  KS*SUPPORTABILITY  +  KM*MANAGEMENT  +  KC*COST 

PROBABILITY  VARIABLES  ane.  expressions  for  the  probabilities  for 
the  technical  area..  These;  expressions,  allow  the  u^.er  to  conduct 
sensitivity  analyses  on  the  technical  area.  Listed' below  are  the 
nominal  values.  Interpreting,  P_TECHB_A  is  the  likelihood  of 
obtaining  a  technical  blue  for  Contractor  A. 

P  TECHB  A  »  0,25000  P  TECHG  A  =■  0.40000 
P  TECHG  B  »  0.50000  P''TECHB~C  »  0.15000 
P~TECHG“c  »  0.7000,0  P^TECHB^D  «  0.50000 

PARAMETER  VALUES  are  expressions  for  scaling  or  importance 
weights  for  the  utility  function.  Once  again,  expressions  enable 
the  user  to  conduct  sensitivity  analyses.  Listed  below  are  the 
nominal  values.  Interpreting,  KT  is  the  scaling  weight  for  the 
technical  area. 

KT  »  0.400  KS  =  0.250 
KM  a  0.200  KC  a  0.150 

II.  Utility  Values.  The  following  utility  values  were  obtained 
from  individual  utility  curves  for  technical,  supportability, 
management,  and  cost  areas..  Since  blue  is  the  most  preferred 
outcome,  the  utility  of  a  blue  is  one  or  U(blue)  a  i,  while  red 
is  the  least  preferred  outcome,  therefore,  the  U(red)  *  0.  The 
utility  for  a  green  or  a  yellow  outcome  is  between  the  U(blue) 
and  U (green).  Utilities  represent,  the  risk  attitude  of  a 
decision  maker.  In  the  case  study,  we  assumed  the  decision  maker 
was  risk  averse.  In  fact,  we  assumed  the  decision  maker  became 
more  risk  averse  as  he/she  moved  from  cost  to  management  to 
supportability  to  technical.  This  coincides  with  importance  of 
the  areas,  i.e.,  cost  is  the  least  important  while  technical  is 
the  most  important.  Below,  is  a  listing  of  utilities. 
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Technical 


U (blue) 

= 

1.00 

U (green) 

= 

0.95 

U (yellow) 

= 

0.85 

U(red) 

= 

0.00 

Supportability 

U(blue) 

= 

1.00 

U (green) 

0.90 

U (yellow) 

= 

0.50 

U (red) 

= 

0.00 

Management 

U (blue) 

=: 

1.00 

'J(  green) 

= 

0.80 

U (yellow) 

0.40 

U(red) 

ss 

0.00 

Cost 

U(blue) 

= 

1.00 

U (green) 

0.65 

U (yellow) 

= 

0.35 

U(red) 

= 

0.00 

III.  Probability  Assessments.  The  probability  assessments  were 
obtained  at  the  item  level,  see  Appendix  1.  This  was 
accomplished  by  assuming  past  performance  was  three  times  as 
important  as  soundness  of  approach,  understanding  the  problem, 
and  compliance  with  requirement.  Then,  we  counted  the  number  of 
specific  colors  (blue,  green,  yellow,  and  red)  and  approached  the 
probabilities  from  a  relative  frequency  stand  point  rather  than  a 
Bayesian  view  point.  In  practice,  most  uncertainties  are 
subjective  judgments  of  a  decision  maker's  belief. 

IV.  Generic  Decision  Tree.  The  generic  decision  tree  represents 
a  graphical  display  of  the  source  selection  case  study.  The  case 
study  was  inputted  into  EXPRESSION  TREE,  thereby  enabling  the 
user  to  compute  the  recommended  course  of  action  and  to  ask  what 
if  questions.  This  was  accomplished  rather  painlessly  and 
required  the  input  of  18  nodes.  The  nodes  included  one  decision 
node,  sixteen  chance  nodes,  and  one  end  node.  The  decision  node 
is  represented  as  a  square  and  has  four  contractors  or 
alternatives.  The  chance  nodes  are  circles  and  are  the 
uncertainties  for  technical,  supportability,  management,  and  cost 
areas  and  have  possible  outcomes  of  blue,  green,  or  yellow.  The 
decimals  included  in  the  generic  decision  tree  represent  the 
likelihoods  of  the  various  outcomes.  The  diamond  is  the  end  node 
which  is  an  algebraic  expression  representing  an  additive  utility 
function.  The  numbers  inside  each  nodes  are  labels  required  for 
entry  into  EXPRESSION  TREE. 
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APPENDIX  3:  AHP  Model 


This  Appendix  provides  the  complete  AHP  model  of  the  case  study. 
Included  in  pages  3-2  through  3-27  is  the  model  represented  in 
hierarchical  form.  Each  page  provides,  at  a  given  level  of  detail, 
some  portion  of  the  model.  Page  3-2  provides  the  "macro”  view  of 
the  model  showing  only  the  most  aggregated  level  of  detail.  The 
remainder  of  Appendix  1  shows  the  lower  levels  of  the  hierarchy  for 
the  "Technical  Area.”  The  rest  of  the  model  (i.e.  the  lower  levels 
of  the  hierarchy  for  the  Supportability,  Management  Capability,  and 
Cost  Areas)  are  similar  but  not  provided  herein. 

Pages  3-28  through  3-50  provide  the  Lotus  spreadsheet,  in  which 
the  AHP  calculations  for  the  synthesis  of  judgements  takes  place. 
It  is  in  this  spreadsheet  that  the  "absolute  measurement”  scheme 
described  in  Section  5  is  implemented  using  the  weights  obtained 
from  EXPERT  CHOICE.  Weights  were  obtained  from  the  EXPERT  CHOICE 
package  for  color  ratings  with  respect  to  each  criterion,  and  the 
for  each  of  the  areas,  items,  and  factors. 

Page  3-50  provides  the  a  summary  of  the  final  evaluations. 
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f>S  OFCISIOM 


:  GOAL  ! 


i  1.  ■}  .  000  i 
!  G  1 . 000 i 


!TECH  : 

I  ( 

!  L  O.SASi 
!  G  0.5f.fti 

j  I 

!-RFl./MATN 
!  L.  0.074 
!  GO.  044. 
!  -PRO/OUAl. 
!  L  0 . 044 
!  G  0.026 
l-SVS  FNGR 

!  L  0 . 4 1  a 

!  GO. 246 
l-FLY  SYfi 
!  L  0.166 

!  G  o..orm 
I -AV TONICS 
!  1.  0.227 

.'  G  0.134 
i-SUPP  SYS 
i  L  0.070 
!  GO. 04 1 


SUPPORT  I  IMANACF  C! 

<  !  ! 

I.  0.260!  !  i  0.002! 

G  0.260!  :  G  0.002! 

_ )  ; _ i 

-ORGAN  ST  I -PROG  MAN 
L  0.078  !  L  0.212 
G  0.020  !  G  0.010 

-MAMAGF  P  !-r-UT  COMP 
1.  0.13.6  !  L  0.106 
G  0.035  !  G  0.010 

-SUP  IMPl.  !-SYS  TFST 
!.  0.22v0  I  L  0.188 

a  o.oao  i  G  0.017 

-l.OaiSTTC  l-MANUFACT 
I.  0.550  !  L  0.324 
0  0.146  !  G  0.030 
i-OiJAl.  ASS 
!  10. 064 

!•  G  0.006 
I -SYS  SAFF 
i  i-  0.049 
!  G  0.004 
'-SECURITY 
!  L  0.057 
i  G  0.005 


!  COST  ! 

I  I 

!  L  0 . 060 1 
I  G  0.060! 

! -REALISM 
I  L  0.327 
!  GO. 020 
I  -RFASONAB 
I  1.  0.413 
1  G  0.025 
! -COMPLETE 
i  L  0.260 
!  G  0.016 
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TECH 


0 


0 

I 

I 


L  0.5SS! 
G  0.588! 

I 


!  ! 

0  0 


IREL/MAIN! 

1  1 

!PRO/QUAL! 

1  1 

1  L  0.074! 

!  L  0.044! 

!  G  0.044! 

{  j 

!  G  0.026! 

1  ! 

!-RELTABIL 

!-PROD  DES 

!  L  0 . 500 

!  L  0.858 

!  G  0.0;?2 

!  G  0.009 

! -MAINTAIN 

1-QUAL  DES 

!  L  0 . 500 

!  L  0.085 

!  G  0.022 

!  G  0.002 
i-MAN  TECH 

!  L  0.379 
!  G  0.010 

i-MAN  RESE 

I  L  0.179 
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